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I.   INTRODUCTION 

A.   DEFENSE  SPENDING  AND  PUBLIC  INTEREST 

In  recent  years,  spending  for  national  defense  has 
increasingly  been  the  focal  point  of  public  scrutiny.  This 
trend  can  be  attributed  to  two  underlying  causes — the  growing 
federal  budget  deficit,  and  the  problem  of  cost  overruns  in 
the  acquisition  of  major  weapons  systems. 

1.   The  Federal  Budget  Deficit 

Between  fiscal  years  1980  and  1987,  the 
Department  of  Defense  annual  Budget  Authority 
almost  doubled,  from  $143  billion  to  $281 
billion,  with  its  total  exceeding  $1  trillion 
during  that  period.  This  sharp  increase 
contributed  to  rising  deficits  and  aroused 
public  concern  over  the  ways  defense  dollars 
were  being  spent.   [Ref.  l:p.  8] 

The  Congressional  Budget  Office's  "Economic  and  Budget 

Outlook:   Fiscal  Years  1992-1996"  indicates  that  since  1986, 

the  defense  budget  has  been  on  a  downward  path.   Total  budget 

authority1  for  1991  was  down  approximately  four  percent  from 

1990.    Moreover,   projections  for  1992  and  1993   indicate 

further  decreases.   [Ref.   3:  p.  84]    As  a  result  of  these 


Budget  authority  is  "the  authority  granted  to  a  federal 
agency  in  an  appropriations  bill,  to  enter  into  commitments  that 
result  in  immediate  or  future  spending."  Budget  authority  is  not 
necessarily  the  amount  of  money  that  an  agency  or  department  will 
spend  during  a  fiscal  year.  Instead,  it  is  merely  the  upper  limit 
on  the  amount  of  new  spending  commitments  it  can  make.  [Ref.  2:p. 
176] 


trends,  it  is  more  important  than  ever  that  the  Department  of 

Defense  (DoD)  manages  its  dollar  resources  effectively. 

One   key   area   in   which   costs   must   be   managed 

effectively  is  the  development  and  procurement  of  weapons 

systems2 . 

During  much  of  the  past  three  decades,  constant  dollar 
unit  costs  for  major  defense  systems  have  grown  much 
faster  than  constant  dollar  total  budgets  for  these 
systems.  The  result  has  been  the  purchase  of  smaller 
quantities  of  new  systems,  delayed  modernization  and 
shrinking  capabilities.   [Ref.  l:p.  10] 

Budget  authority  and  outlays  for  research,  development,  test 

and  evaluation  and  procurement  accounted  for  approximately 

forty  percent  of  the  national  defense  budget  in  1989  [Ref. 

4:p.  A-146].    More  effective  and  efficient  utilization  of 

these  funds  could  result  in  significant  savings  within  the 

DoD,  thus  allowing  scarce  resources  to  be  applied  to  other 

important  requirements. 

2.   Cost  Overruns  in  Weapons  Acquisition 

The   second  underlying  cause  of   increased  public 

scrutiny  of  defense  spending  is  cost  overruns.  The  problem  of 

cost  overruns  in  the  weapons  acquisition  process  has  been  a 

major  source  of  consternation  and  embarrassment  for  the  DoD 

for  many  years.     Numerous   researchers   and  presidential 


:  The  term  weapon  system  normally  refers  to  the  major  item  of 
equipment  and  the  subsystems,  logistical  support,  software, 
construction  and  training  needed  to  operate  and  support  it  [Ref. 
l:p.  9].  For  purposes  of  this  thesis,  weapon  system  refers  only  to 
the  major  item  of  equipment. 


commissions  during  the  past  thirty  years  have  concluded  that 
tens  of  billions  of  dollars  per  year  could  be  saved  by 
improving  the  acquisition  process  [Ref.  l:p.  32].  "The 
studies  (have)  repeatedly  urged  Congress  and  the  Defense 
Department  to  correct  five  basic  deficiencies: 


1.  Setting  requirements  for  the  most  sophisticated 
systems  available,  often  irrespective  of  cost; 

2 .  Changes  in  program  and  contract  requirements  caused  by 
changes  in  military  user  preferences,  leading  to 
annual  or  more  frequent  changes  in  program  funding 
levels,  initiated  by  Congress  and  DoD  itself; 

3 .  Lack  of  incentives  for  contractors  and  government 
personnel  to  reduce  program  costs; 

4.  Failure  to  develop  sufficient  numbers  of  military  and 
civilian  personnel  with  training  and  experience  in 
business  management  and  in  dealing  with  industrial 
firms  to  oversee  the  development  and  production  of 
enormous,  highly  technical  industrial  programs;  and 

5.  Underestimated  schedules  and  costs  of  major  programs, 
distorting  the  decision-making  process  for  the 
allocation  of  the  national  budget."  [Ref.  l:p.  32] 


While  progress  has  been  made  in  each  of  the  areas 
identified  above,  there  remains  much  room  for  improvement. 
Major  defense  procurement  programs  have  repeatedly  experienced 
significant  unanticipated  schedule  delays  and  cost  increases. 
More  than  ninety  percent  of  these  programs  exceed  initial  cost 
estimates  and,  in  the  majority  of  cases,  the  average  increase 
in  cost  has  been  more  than  fifty  percent,  excluding  the 
effects  of  quantity  changes  and  inflation.   [Ref.  l:p.  32-33] 


A  cost  overrun  occurs  when  the  actual  cost  of  a 
program  exceeds  the  estimated  cost.  Cost  overruns  typically 
occur  when:  fair  initial  cost  estimates  are  made  but 
subsequent  actual  costs  are  poorly  managed  and  controlled;  or 
actual  costs  are  well-managed  but  initial  cost  estimates  were 
unrealistic.   [Ref.  5:p.  1] 

There  are  various  reasons  why  initial  cost  estimates 
may  be  unrealistic,  particularly  unrealistically  low.  For 
example,  institutional  incentives  may  exist  both  within  DoD 
and  at  government  contractors  to  underestimate  costs  initially 
in  order  to  get  a  program  approved  and  started.  A  second 
possible  reason  is  that  processes,  techniques  or  tools  for 
creating  cost  estimates  may  be  weak  and  provide  misleading 
cost  forecasts.  This  thesis  addresses  a  technical  question 
related  to  the  latter  issue.  In  particular,  this  paper 
examines  the  problem  of  low/unrealistic  cost  estimates  for 
major  weapons  programs  by  analyzing  the  performance  of 
alternative  cost  estimation  models. 

B.   COST  ESTIMATION  MODELS 

Broadly,  cost  estimation  models  fall  into  two  categories. 
First,  Cost  Estimating  Relationships  (CERs)  attempt  to  explain 
or  predict  the  cost  of  a  "standard"  unit  of  an  item  to  be 
manufactured  or  procured  in  terms  of  variables  reflecting 
qualities,  attributes  or  characteristics  of  the  item.  For 
example,  aircraft  costs  may  be  modeled  in  terms  of  speed  or 


payload.  Second,  Cost  Progress  Models  attempt  to  explain  or 
predict:  changes  in  unm  cost  of  items  over  the  life  of  a 
production  or  procurement  program  in  terms  of  changes  in  the 
circumstances  surrounding  production  or  acquisition.  For 
example,  costs  may  be  expected  to  depend  on  the  numbers 
acquired  and  the  production  rate  and  thus  unit  costs  may  be 
modeled  in  terms  of  such  variables.  The  analysis  in  this 
thesis  will  focus  on  the  latter  type  of  model,  cost  progress 
models. 

1.   The  Learning  Curve 

The  most  commonly  used  cost  progress  model  is  the 
learning  curve. 


Learning  curves  have  gained  widespread  acceptance  as  a 
tool  for  planning,  analyzing,  explaining  and  predicting 
the  behavior  of  the  unit  cost  of  items  produced  from  a 
repetitive  production  process  [Ref.  5:p.  1]. 


Although  learning  curves  were  originally  developed  and  applied 
to  predict  cost  and  time  requirements  for  the  construction  of 
ships  and  aircraft  during  World  War  II,  they  have  since  been 
applied  in  many  other  manufacturing  and  non-manufacturing 
settings.  The  learning  curve  phenomena  was  first  reported  by 
T.  P.  Wright  in  the  Journal  of  Aeronautical  Sciences  in  193  6. 
Wright  observed  that,  as  the  quantity  of  units  manufactured 
doubles,  the  number  of  direct  labor  hours/cost  associated  with 
the  production  of  an  individual  unit  decreases  at  a  uniform 
rate.   Moreover,  the  uniform  rate  of  learning  is  peculiar  to 


the  manufacturing  process  being  observed. 
2.   Alternative  Cost  Progress  Models 

It  is  generally  acknowledged  that  other  factors,  in 
addition  to  cumulative  quantity,  influence  unit  cost  and  that 
the  simple  learning  curve  does  not  provide  a  fully  adequate 
description  of  cost  behavior.  As  a  result,  prior  research  has 
attempted  to  improve  the  simple  learning  curve  model  by 
including  additional  variables.   [Ref.  5:p.  2] 

There  are  now  multiple  approaches  and  models  available 
for  estimating  the  costs  of  acquisition  programs.  Two  of  the 
most  commonly  used  cost  progress  model  types  are  the  learning 
curve  and  the  production  rate  adjustment  model.  Other  model 
types  include  the  plateau  model,  the  Stanford-B  model,  the  De 
Jong  model  and  the  S-model.  These  models  are  differentiated 
by  the  variables  included  and  the  underlying  assumptions. 

C.   PURPOSE 

It  is  unclear  at  present  which  model  type  is  most 
appropriate  for  predicting  costs  under  various  manufacturing 
conditions.  The  purpose  of  this  thesis  is  to  evaluate  the 
quality  of  cost  estimates  produced  by  each  of  four  cost 
progress  models — a  random  walk  model,  a  learning  curve  model, 
a  production  rate  model,  and  a  model  incorporating  both 
learning  curve  and  production  rate  effects.  In  conducting 
this  evaluation,  emphasis  will  be  place  on  assessing  the  level 
of  bias  associated  with  each  of  these  models. 


D.   RESEARCH  QUESTIONS 

This  thesis  will  address  the  following  research  questions: 

1 .  Primary  Research  Question 

What  is  the  bias  exhibited  by  available  cost  progress 
models  when  predicting  the  future  unit  cost  of  weapons  systems 
acquired  through  a  continuing  acquisition  program? 

2 .  Subsidiary  Research  Questions 


a.  Are  the  various  available  cost  progress  models 
comparable  in  terms  of  bias? 

b.  Do  particular  models  result  in  less  biased 
estimates  under  certain  conditions? 

c.  What  are  those  conditions  that  affect  the 
performance  of  the  models? 

d.  Can  guidelines  be  established  for  determining  when 
(under  what  conditions  or  circumstances)  it  is 

most  appropriate  to  use  a  particular  model  type? 


E.   SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

There  are  a  number  of  different  criteria  which  can  be 
examined  in  order  to  assess  various  aspects  of  model 
performance.  Two  of  these  criteria  which  are  particularly 
important  are  the  accuracy  and  the  level  of  bias  associated 
with  a  particular  model.  Accuracy  refers  to  the  degree  of 
error  in  a  model's  prediction,  without  regard  to  the  direction 
of  the  error.  Bias  refers  to  both  the  direction  and  the 
magnitude  of  error.  It  indicates  whether  predictions  made 
using  a  particular  model  underestimate  or  overestimate  actual 
cost.   The  focus  of  this  thesis  will  be  limited  to  an  analysis 


of  the  bias  associated  with  the  various  models  tested.  One 
purpose  of  the  study  is  to  either  confirm  or  disconfirm  the 
results  of  an  earlier  simulation  study  by  Moses  ( Learning 
Curve  and  Rate  Adjustment  Models:  An  Investigation  of  Bias) 
and  to  determine  whether  or  not  those  results  hold  when 
testing  real  world  data.  A  second  purpose  is  to  extend  the 
analysis  of  bias  to  a  larger  set  of  models. 

F.   ORGANIZATION  OF  THESIS 

The  remainder  of  this  thesis  is  organized  into  four 
chapters.  Chapter  II  provides  a  review  of  the  literature 
dealing  with  various  cost  progress  models.  Chapter  III 
provides  a  description  of  the  sample,  data  and  measures  used 
to  conduct  the  study.  Chapter  IV  provides  an  analysis  of  the 
results.  Finally,  chapter  V  summarizes  the  research  findings 
and  suggests  directions  for  future  research. 


II.  REVIEW  OF  PREVIOUS  RESEARCH 

Although  the  progress  or  learning  curve  technique  of  cost 
estimation  was  discovered  prior  to  World  War  II,  several 
decades  passed  before  statistical  studies  of  this  phenomena 
could  be  readily  conducted.  This  situation  can  be  attributed 
to  two  underlying  problems:  (1)  the  available  data  were 
frequently  too  sparse  to  support  statistical  analyses,  and; 
(2)  the  sheer  volume  of  calculations  and  lack  of  powerful 
computers  meant  that  many  laborious  hours  were  required  to 
perform  operations  which  can  today  be  performed  in  a  few 
minutes.  This  second  problem  had  a  strong  inhibiting  effect 
on  researchers.  As  a  result,  it  was  not  until  the  1950' s  that 
significant  statistical  studies  of  the  learning  curve 
phenomena  began  to  be  undertaken.  [Ref.  6:p.  8] 

This  chapter  reviews  the  literature  dealing  with 
development  of  the  learning  curve  and  alternative  rate 
adjustment  models.  In  particular,  it  summarizes  the  findings 
of  some  of  the  major  studies  which  have  been  conducted  in  an 
effort  to  evaluate  the  relative  performance  of  these  models. 
The  discussion  is  organized  chronologically  into  the  following 
groupings:  research  prior  to  1970,  research  during  the 
seventies,  research  during  the  eighties,  and  research  during 
the  nineties. 


A.   SURVEY  OF  RESEARCH  PRIOR  TO  197  0 

1.  Hirsch 

In  1952,  Hirsch  [Ref.  7]  published  the  results  of  a 
five-year  study  of  a  large  United  States  machine  builder.  The 
purpose  of  the  study  was  to  examine  the  relationship  between 
labor  requirements  and  production  volume.  Lot  size  was  used 
as  a  measure  of  the  rate  of  production  based  on  the  existence 
of  stable  production  lot  intervals.  In  addition,  practically 
no  changes  in  management  or  plant  and  equipment  occurred 
during  the  period  of  study.  Based  on  the  results  of  his 
study,  Hirsch  concluded  the  relation  between  direct  labor 
requirements  and  lot  size  was  of  little  consequence  in  the 
machining  and  assembling  processes. 

2 .  Cochran 

In  1960,  E.  B.  Cochran  [Ref.  8]  published  an  article 
in  which  he  conducted  a  careful  examination  of  the  basic  cost 
function — the  learning  curve — by  studying  specific 
manufacturing  conditions  and  parameters  which  relate  them  to 
cost  trends.  The  purpose  of  this  study  was  to  probe  various 
learning  curve  applications  in  an  effort  to  develop  new 
concepts  of  learning  curve  analysis  and  revised  applications. 
Based  on  the  results  of  his  study,  Cochran  proposed 
refinements  to  the  basic  learning  curve  concept.  In 
particular,  he  suggested  that  learning  may  not  necessarily 
occur  exactly  once  per  unit  but  instead,  may  occur  either 
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faster  or  slower.  For  example,  when  planning  airframe  costs 
for  a  four-engine  aircraft,  it  may  be  appropriate  to  consider 
each  engine  pod  as  a  unit  of  learning  rather  than  simply 
consider  the  aircraft  as  a  unit  of  learning. 

Cochran  provided  guidelines  for  accurately  identifying 
the  unit  of  learning  and  identified  other  factors,  in  addition 
to  worker  learning,  that  would  affect  the  rate  of  cost 
reduction  (e.g.,  tooling,  supervision,  parts  design  and 
shortages) .  Moreover,  he  demonstrated  how  changing  rates  of 
learning  and  task  changes  may  result  in  both  shifts  of  the 
learning  curve  and  non-linear  learning.  One  of  the  most 
significant  findings  presented  by  Cochran  was  that: 


Any  change  in  learning  rate  is  equivalent  to  a  change  in 
the  unit  at  which  standard  cost  is  reached.  And  this  in 
turn  generates  a  major  shift  in  the  entire  level  of  cost 
[Ref.  8:p.  319]. 


As  a  result  of  this  finding,  Cochran  concluded  "that 
the  determination  of  learning  rate  is  of  major  significance  in 
forecasting  and  controlling  costs."  [Ref.  8:p.  319] 

Cochran  indicated  that  the  shape  of  the  learning  curve 
can  be  critical  in  the  first  100  units  and  pointed  out  the 
fact  that  there  is  a  wide  range  of  error  in  straight  line 
curves.  Accordingly,  he  suggested  that  an  S-curve  pattern  may 
be  more  appropriate  than  the  usual  linear  learning  curve. 
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3 .   Alchian 

In  1963,  Alchian  [Ref.  9]  published  the  results  of  a 
1949  study  conducted  for  the  RAND  Corporation.  The  purpose  of 
this  study  was  to  examine  the  similarity  of  airframe 
manufacturing  progress  functions  among  various  airframe 
manufacturers.  Statistical  tests  of  the  similarity  of  the 
functions  among  various  airframe  manufacturers  were  performed 
using  World  War  II  data.  In  addition,  the  reliability  of 
predictions  made  with  these  curves  was  assessed. 

The  results  of  this  study  indicated  the  progress 
functions  differed  among  various  airframe  types  and 
manufacturing  facilities  both  in  the  amount  and  rate  of  change 
of  required  direct  labor  per  pound  of  airframe.  Alchian 
suggested  that,  for  practical  purposes,  the  use  of  an  average 
of  individual  progress  functions  may  be  appropriate.  By 
applying  this  procedure  to  22  airframes  produced  at  different 
facilities,  the  average  production  error3  was  found  to  be 


Production  error  was  defined  as  follows: 

Predicted  Manhours  -  Actual  Manhours 


Production  Error 


Actual  Manhours 


Direct  labor  requirements  (manhours)  for  the  first  1000  planes  were 
predicted  for  22  aircraft  model — facility  combinations  using  both 
an  industry  progress  curve  and  an  airframe  type  progress  curve. 
The  percentage  error  resulting  from  the  use  of  each  of  these  curves 
was  then  computed  for  each  model — facility  combination  using  the 
equation  described  above.  Next,  the  weighted  average  error  per 
facility  (weighted  by  actual  manhours)  associated  with  the  use  of 
the  industry  and  airframe  progress  curves  was  computed  for  each  of 
the  four  major  aircraft  model  groups  examined:  bombers,  fighters, 
trainers  and  transports.  Based  on  these  figures,  the  weighted 
average  error  per  facility  for  all  facilities  was  computed. 
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approximately  25  percent.  This  same  result  was  obtained  for 
the  entire  output  of  any  particular  airframe  produced  in  one 
facility.  Specific  curves  fitted  to  the  past  performance  of 
a  particular  manufacturing  facility  resulted  in  margins  of 
error  of  approximately  20  percent. 

Alchian  examined  alternative  relationships  between 
direct  labor  per  pound  of  airframe,  cumulative  number  of 
airframes,  time  and  rate  of  production.  "The  results  cast 
doubts  on  any  of  the  alternatives  being  better  fits  than  the 
usual  progress  curve."  [Ref.  9:p.  692] 

B.   SURVEY  OF  RESEARCH  DURING  THE  SEVENTIES 
1.   Linder  and  Wilbourn 

Models  which  relate  costs  to  various  cost-driving 
features  or  parameters  (i.e.,  physical/performance  parameters 
of  the  weapons  system)  typically  result  in  reasonable 
estimates  of  future  recurring  unit  procurement  costs  [Ref. 
10:p.  277].  Nevertheless,  Linder  and  Wilbourn  [Ref.  10] 
suggested  that  in  addition  to  these  parameters,  various 
characteristics  of  the  procurement  program  itself  (e.g., 
competitive  versus  sole-source  procurement,  single-year  versus 
multiple-year  buys,  and  low  versus  high  production  rates) 
represent  cost-driving  features  which  should  be  accounted  for 
in  the  cost  estimating  procedure. 
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Linder  and  Wilbourn  investigated  the  effect  of 
production  rate  on  recurring  missile  unit  procurement  costs. 
In  particular,  they  developed  two  models  to  examine  how 
production  rate  influences  the  position  and/or  slope  of  the 
recurring  missile  hardware  cost  improvement  curve.  The  first 
model  formulated  unit  recurring  cost  as  a  function  of  a 
constant  "annual"  production  rate.  The  second  model 
formulated  unit  recurring  cost  as  a  function  of  a  variable 
annualized  production  rate. 

These  models  were  developed  based  on  an  analysis  of 
the  impact  of  production  rate  changes  on  direct  and  indirect 
costs.  Linder  and  Wilbourn  reasoned  that  higher  production 
rates  would  result  in  lower  fixed  costs  per  unit.  Moreover, 
high  production  rates  were  expected  to  lead  to  a  smaller 
percentage  increase  in  indirect  costs  than  direct  costs.  As 
a  result,  lower  overhead  rates  should  be  applied  to  direct 
costs.  Based  on  the  combined  effects  of  lower  fixed  costs  per 
unit  and  lower  applied  overhead  rates,  Linder  and  Wilbourn 
concluded  that  unit  costs  would  be  reduced  at  higher 
production  rates,  ceteris  paribus.  Moreover,  the  cost 
improvement  curve  associated  with  a  high  production  rate  was 
expected  to  lie  below  a  cost  improvement  curve  associated  with 
a  lower  production  rate. 

In  assessing  the  expected  effect  of  production  rate  on 
the  cost  improvement  curve  slope,  the  researchers  made  the 
following  observations: 

14 


1.  At  high  production  rates,  average  fixed  direct  costs  are 
reduced  and  variable  direct  costs  constitute  a  larger 
portion  of  total  direct  costs. 

2.  As  direct  costs  fall,  indirect  costs  are  reduced  at  a 
slower  rate  such  that  overhead  rates  increase. 


These  two  phenomena  have  opposite  affects  on  the  slope  of 
the  cost  improvement  curve.  The  first  one  will  tend  to 
increase  the  slope  of  the  curve  due  to  the  influences  of 
learning  and  other  related  effects  on  variable  directs  costs 
at  a  high  production  rate.  The  latter  phenomena  will  tend  to 
flatten  the  curve  as  production  increases.  As  a  result,  "the 
net  effect  on  the  cost  improvement  curve  slope  depends  on  the 
relative  amounts  of  direct  and  indirect  costs  per  unit  as  well 
as  the  proportion  of  each  cost  category  which  can  be 
considered  as  fixed  or  variable."  [Ref.  10:p.  280] 

Based  on  the  results  of  their  analyses,  Linder  and 
Wilbourn  reached  the  following  conclusions: 


1.  Ceteris  paribus,  higher  production  rates  result  in  lower 
unit  recurring  costs  at  each  production  quantity. 

2.  Doubling  the  production  rate  lowers  average  unit  costs  by 
approximately  three  to  seven  percent  for  the  quantities 
examined. 

3.  Changing  the  production  rate  has  only  a  slight  influence 
on  the  slope  of  the  unit  cost  improvement  curve. 

4.  The  effects  discussed  above  are  relatively  insensitive  to 
changes  in  the  models'  parameter  values.  [Ref.  10: p.  3  00] 
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2.   RAND  Studies 

During  the  1970 's,  the  RAND  Corporation  conducted  a 
number  of  studies  which  examined  the  relationship  between 
production  rate  and  airframe  costs.  The  specific  objectives 
and  results  of  two  of  these  studies  are  summarized  below. 

a.    Large,    Hoffmayer  and  Kontrovich 

In  1974,  the  results  of  a  study  by  Large,  Hoffmayer 
and  Kontrovich  [Ref .  11]  were  published.  "The  purpose  of  this 
study  was  to  investigate  the  nature,  magnitude  and  causes  of 
the  influence  of  production  rate  on  unit  cost."  [Ref.  ll:p. 
iii]  Based  on  the  assumption  that  production  rate  and  unit 
cost  vary  inversely,  the  researchers  sought  to  develop  an 
estimating  model  to  express  the  relationship  for  various 
elements  of  cost. 

The  results  of  the  analysis  suggested  the  effect  of 
production  rate  on  manufacturing  labor,  manufacturing 
materials,  tools  and  engineering  could  not  be  predicted  with 
confidence.  In  any  specific  case,  the  effect  depended  on  a 
number  of  factors  including  how  rate  changes  were  achieved, 
the  availability  of  suppliers,  the  local  labor  supply, 
management  policy,  the  timing  of  rate  changes,  plant  capacity, 
and  plant  backlog.  The  only  element  of  cost  which  was  found 
to  clearly  be  a  function  of  production  rate  was  overhead. 

Based  on  their  findings,  the  researchers  concluded 
that  the  influence  of  production  rate  on  aircraft  cost  could 
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not  be  predicted  with  any  degree  of  confidence.  Each  case 
should  be  examined  separately  and  in  detail  to  assess  the 
effect  of  rate.  In  addition,  they  suggested  that  in  advanced 
planning  studies,  rate  effects  in  aircraft  production  programs 
can  be  ignored  because  they  are  far  outweighed  by  other 
uncertainties.  As  a  result,  they  indicated  that  a  model  that 
does  not  explicitly  consider  rate  may  be  preferable  for 
advance  planning  purposes. 

b.    Large,    Campbell   and  Cates 

This  study,  published  in  1976,  attempted  to  derive 
improved  parametric  equations  for  estimating  the  acquisition 
cost  of  aircraft  airframes.  Earlier  RAND  studies  had 
indicated  variations  in  cost  among  different  airframes  were 
best  explained  by  the  quantity  produced  and  aircraft 
characteristics,  i.e. ,  airframe  unit  weight  and  maximum  speed. 
Large,  Campbell  and  Cates  [Ref.  12]  were  unable  to  identify 
additional  characteristics  that  would  make  an  estimating  model 
more  flexible  and,  hence,  better  able  to  deal  with 
characteristics  peculiar  to  individual  aircraft.  None  of  the 
independent  variables  considered  significantly  improved  the 
reliability  of  estimates  obtained  using  only  weight,  speed  and 
cumulative  quantity.  As  a  result,  they  suggested  that  future 
research  which  examines  the  influence  of  program 
characteristics  on  program  cost  may  be  more  productive. 
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3 .   Smith 

The  purpose  of  this  study  was  to  develop  and  test  a 
procedure  to  assess  the  effect  of  production  rate  changes  on 
the  direct  labor  requirements  for  production  of  additional 
airframes.  Smith  [Ref.  13]  proposed  a  cost  model  to  express 
direct  labor  hours  required  as  a  function  of  cumulative 
production  and  production  rate.  Two  approaches  to  expressing 
the  production  rate  variable  were  examined:  a  lot  average 
delivery  rate,  and  a  lot  average  manufacturing  rate.  Data 
from  three  airframe  production  programs — the  F-4  program,  the 
F-102  program,  and  the  KC-135  program — were  used  to  construct 
data  sets.  These  data  sets  were  then  examined  in  the 
cumulative  production  and  production  rate  cost  model  using 
regression  analysis. 

Production  rate  was  found  to  be  an  important  factor  in 
the  cost  of  airframe  production,  although  its  effect  was 
subordinate  to  that  of  cumulative  production.  The  study 
demonstrated  empirically  that  production  rate  can  be  an 
important  predictor  of  variation  in  unit  direct  labor 
requirements.  In  addition,  the  results  suggested  an  increase 
in  rate  up  to  plant  capacity  can  lead  to  a  decrease  in  unit 
labor  requirements. 
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C.   SURVEY  OF  RESEARCH  DURING  THE  EIGHTIES 
1.   Crouch 

The  purpose  of  this  study  was  to  investigate  possible 
sources  of  bias  in  the  slopes  of  progress  functions  which  are 
conventionally  estimated  using  the  unit  cost  progress 
function.  Crouch  [Ref.  14]  asserted  that  given  the  fact  that 
unit  costs  are  a  function  not  only  of  cumulative  output  but 
also  of  the  rate  of  output  per  time  period,  the  conventional 
unit-cost  progress  function  has  omitted  variables.  Under 
these  circumstances,  the  use  of  ordinary  least  sguares 
regression  may  introduce  specification  bias  into  the 
estimates.   Crouch  confirmed  this  situation  mathematically. 


When  a  variable  from  the  true  relation  is  omitted,  a  part 
of  its  influence  in  explaining  the  movements  of  the 
dependent  variable  is  captured  by  the  independent 
variables  which  are  included.  When  the  omitted  variable 
is  not  correlated  with  any  of  the  independent  variables, 
the  coefficients  of  the  included  variables  are  not  biased. 
[Ref.  14:p.  42] 


Crouch  conducted  a  pilot  study  to  investigate  the 
existence  of  bias  when  progress  functions  are  estimated  in  the 
conventional  manner.  Unit-cost  data  (on  an  annual  average 
basis)  in  constant  dollars  for  ten  components  of  the  Hawk 
missile  were  used.  The  results  of  this  study  indicated  that 
when  progress  functions  are  estimated  in  the  conventional 
manner,  biased  estimates  of  the  slopes  may  be  obtained  in  a 
significant  number  of  cases.    In  particular,  the  results 
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indicated  that  when  returns  to  scale  are  constant,  the 
estimated  learning  curve  exponent  obtained  from  the 
conventional  progress  function  will  be  unbiased.  However, 
when  the  returns  to  scale  are  not  constant,  the  estimated 
learning  curve  exponent  obtained  from  the  conventional 
progress  function  will  be  biased.  Negative  bias  will  occur 
when  returns  to  scale  are  increasing;  positive  bias  will  occur 
when  returns  to  scale  are  decreasing. 
2 .   Smith 

The  objective  of  this  study  was  to  examine  the  impact 
of  production  rate  changes  on  the  unit  cost  of  weapons 
systems.  Smith  [Ref.  15]  provided  a  summary  of  the 
significant  research  on  the  relationship  between  production 
rate  and  weapon  system  cost.  In  doing  so,  he  analyzed  the 
various  findings  and  conclusions  and  assessed  their 
applicability.  The  results  of  his  research  review  indicated 
that  only  rather  weak  conclusions  could  be  drawn  from  the 
existing  state  of  knowledge.  The  principal  findings  were: 
production  rate  affects  unit  costs  but,  in  most  cases,  not  as 
strongly  as  the  learning  (cumulative  guantity)  effect,  and; 
the  rate  effect  varies  with  the  weapon  system.  Smith 
identified  four  principal  cost-rate  models — Womer  [Ref.  16], 
Washburn  [Ref.  17],  Linder  and  Wilbourn  [Ref.  10],  and  Fazio 
and  Russell  [Ref.  18].  Each  of  these  models  differed  in  the 
concept  of  rate,  the  number  of  parameters  to  estimate,  and  the 
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range  of  applicable  programs.  Nevertheless,  none  of  them  was 
considered  suitable  for  use  by  top  level  budget  planners. 
[Ref.  15:p.  ii] 

Case  studies  were  conducted  using  production  data  for 
six  missile  systems.  These  systems  possessed  a  wide  range  of 
production  characteristics  ranging  from  low  volume,  labor 
intensive  to  high  volume,  highly  automated  production.  The 
results  of  the  empirical  research  supported  the  belief  that 
under  program  stretch-out,  the  most  important  contributor  to 
increased  unit  costs  is  an  increased  overhead  allocation.  In 
addition,  the  idea  that  labor  inefficiency  is  often  a 
relatively  unimportant  factor  in  rate  adjustments  was 
supported.  [Ref.  15:pp.  43-44]  Based  on  the  results  of  the 
case  studies,  Smith  concluded  that: 


1.  A  simple  rate-sensitive  model  which  focuses  on  the  effect 
of  rate  changes  on  overhead  is  appropriate  for  the 
programming  and  budgeting  phases. 

2 .  Long-range  planners  should  disregard  rate  behavior  and 
focus  only  on  military  reguirements.  In  long-term 
planning,  production  rates  and  their  effects  are  both 
unpredictable  and  much  less  important  than  other  more 
fundamental  considerations.  [Ref.  15: p.  46] 


3 .   Balut 

Standard  use  of  learning  curve  theory  involves  an 
implicit  assumption  that  overhead  is  100  percent  variable  with 
direct  costs.  However,  plant  overhead  is  actually  comprised 
of  three  components — variable  overhead,  fixed  overhead,  and 
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semi-fixed  overhead.  Variable  overhead  costs  vary  with  the 
activity  rate.  They  include  production-related  indirect  costs 
that  are  tied  to  the  number  of  direct  laborers  working  in  the 
plant  and  the  number  of  units  being  produced.  Fixed  overhead 
costs  do  not  vary  with  the  activity  rate  and  are  fixed  in  the 
short-term  (e.g.,  depreciation,  insurance,  rent,  security). 
Semi-fixed  overhead  is  indirect  expenses  that  are  partially 
fixed  and  partially  variable  such  as  utilities.  Semi-fixed 
overhead  costs  are  typically  gathered  and  reported  as  pools; 
consequently,  the  fixed  and  variable  portions  are  not 
discernable.  [Ref.  19:pp.  63-65] 

The  costing  of  alternative  aircraft  procurement 
quantities  and  rates  within  the  Office  of  the  Secretary  of 
Defense  (Program  Analysis  and  Evaluation)  is  a  two  step 
process  which  considers  the  heterogeneous  nature  of  overhead. 
First,  new  average  unit  prices  are  derived  for  each  lot, 
consistent  with  new  lot  quantities,  using  the  learning  curve. 
Then,  in  order  to  correct  for  the  erroneous  underlying 
assumption  in  step  one,  prices  are  adjusted  to  reflect  the 
redistribution  of  fixed  overhead  resulting  from  a  change  in 
the  production  rate.  This  second  step  is  referred  to  as  rate 
adjustment.  [Ref.  19:p.  65] 

The  objective  of  Balut's  [Ref.  19]  study  was  to 
evaluate  the  rate  adjustment  model  used  by  the  Office  of  the 
Secretary  of  Defense  by  comparing  its  predictions  to  actual 
contractor  performance.    An  improved  version  of  the  model 
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derived  using  actual  contractor  data  was  presented.   Balut 
illustrated  the  use  of  this  improved  model  for  situations  when 
the  contractor  has  only  one  program,  and  for  situations  when 
the  contractor  has  other  ongoing  programs. 
4.   Bemis 

Unit  costs  for  weapons  systems  have  traditionally  been 
projected  using  the  experience  curve.  This  method  expresses 
the  projected  unit  cost  as  a  function  of  cumulative  guantity 
produced,  regardless  of  the  production  rate.  Prior  research 
into  the  production  rate-cost  relationship  for  weapons  systems 
indicates  that  unit  cost  varies  significantly  as  a  function  of 
production  rate.  These  variations  are,  to  a  great  extent,  due 
to  the  amortization  of  fixed  overhead.  [Ref.  20:pp.  84-85] 

Bemis  [Ref.  20]  proposed  a  method  for  estimating 
rate/cost/guantity  relationships  using  system  specific  cost 
estimates.  Only  unit-fly-away  costs  were  considered.  The 
input  data  for  the  model  was  historical  rate/cost/guantity 
data  for  ongoing  programs,  and  contractor  or  in-house 
estimates  for  new  programs.  An  eguation  was  derived  by 
regressing  unit  cost  on  cumulative  guantity  and  production 
rate.  In  most  of  the  cases  analyzed  in  this  study,  a  high 
multiple  correlation  coefficient  (greater  than  0.9)  was 
obtained. 

Bemis  found  that  when  the  production  rate  was  stable, 
the  experience  curve  method  and  the  rate/cost/guantity  method 
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generated  identical  unit  cost  estimates.  However,  when  the 
production  rate  was  variable,  lower  unit  costs  were  associated 
with  higher  production  rates,  and  higher  unit  costs  were 
associated  with  lower  production  rates. 

Bemis  suggested  the  rate/cost/quantity  model  could  be 
an  invaluable  tool  for  approaching  "what  if"  questions  in  the 
planning  and  budgeting  process.  Moreover,  he  suggested  this 
model  offered  users  a  means  for  readily  assessing  the  cost 
effects  of  program  stretchouts,  the  costs  of  maintaining  a 
warm  production  base,  and  the  probable  effects  of  program 
acceleration. 

D.   SURVEY  OF  RESEARCH  DURING  THE  NINETIES 
1.   Boger  and  Liao 

In  an  effort  to  reflect  the  effect  of  production  rate 
on  the  cost  of  weapons  systems,  researchers  have  proposed  a 
variety  of  adjustments  to  weapons  systems  cost  models.  The 
most  popular  approach  has  been  to  augment  the  traditional 
learning  curve  by  adding  a  rate  term.  The  resulting  learning 
curve  is  referred  to  as  a  rate  adjustment  model.  [Ref.  21: p. 
32]  Boger  and  Liao  [Ref.  21]  examined  the  effects  of 
different  rate  measures  and  cost  structures  on  rate  adjustment 
models  and  illustrated  how  alternative  surrogate  production 
rate  measures  might  lead  to  erroneous  conclusions. 

The  effect  of  production  rate  on  unit  cost  stems  from 
economies  of  scale.    As  production  rates  are  increased, 
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facilities  are  utilized  more  fully  and  greater  specialization 
of  labor  occurs.  Materials  costs  are  reduced  because  the 
increased  volume  of  materials  purchased  results  in  quantity 
discounts.  Finally,  the  increased  production  volume  allows 
fixed  overhead  charges  to  be  spread  over  a  larger  quantity  of 
output.  Together,  these  underlying  effects  work  to  increase 
efficiency  and  lower  production  costs.  [Ref.  21 :p.  83] 

Increases  in  production  rate  are  normally  expected  to 
result  in  lower  unit  costs  due  to  economies  of  scale;  however, 
production  rate  increases  may  also  lead  to  diseconomies  of 
scale.  Such  is  the  case  when  a  plant  is  operating  beyond  its 
efficient  capacity  level.  Under  these  circumstances,  factors 
such  as  over-time  pay,  lack  of  skilled  labor  or  the  need  to 
invest  in  additional  tooling  and/or  facilities  may  lead  to 
inefficiencies  and  increased  unit  costs.  [Ref.  21:pp.  23-24] 

Because  of  the  difficulty  associated  with  measuring 
production  rate,  a  number  of  alternative  surrogate  measures 
have  been  adopted.  The  two  primary  surrogate  production  rate 
measures  are  lot  size  and  annual/monthly  production  quantity. 
Unfortunately,  there  are  weaknesses  associated  with  the  use  of 
each  of  these  measures.   These  weaknesses  are  as  follows: 


1.  Lot  Size:  The  time  required  to  produce  successive, 
comparably-sized  lots  frequently  changes  over  the  life  of 
a  program.  As  a  result,  it  is  unclear  what  is  being 
measured  by  the  lot  size  proxy. 

2.  Production  Quantity/Time  Interval:  If  there  is  a  large 
amount  of  work-in-progress  and  the  production  period  is 
long  compared  to  the  observation  period,  units  produced 
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in  the  following  time  period  will  actually  reflect  work 
performed  in  the  previous  time  period.  This  can  result 
in  substantial  bias  in  estimation. 

3.  Average  Rate  for  Each  Program:  This  approach  may- 
understate  the  effect  of  descriptive  rate  changes.  An 
average  rate  for  each  program  is  usually  used  in  cross- 
sectional  analysis  because  the  production  rate  may  change 
in  a  typical  production  run. 

4.  Cumulative  Quantity:  Cumulative  quantity  is  highly 
correlated  with  each  of  the  surrogate  production  rate 
measures  discussed  above.  As  a  result,  analysts  have 
been  unable  to  separate  statistically  the  effect  of 
learning  and  production  rate.  [Ref.  21:pp.  86-87] 


In  order  to  avoid  some  of  the  difficulties  associated 
with  using  these  surrogate  production  rate  measures,  Boger  and 
Liao  recommended  a  ratio  of  these  measures  be  used.  This 
ratio  should  be  keyed  to  a  base  production  rate.  Adoption  of 
this  approach  offers  a  number  of  advantages: 


1.  Using  a  surrogate  production  rate  ratio  tends  to  mitigate 
the  multicollinearity  problem. 

2.  Using  the  rate  to  which  the  manufacturer  has  tooled  as 
the  base  rate  provides  an  indicator  of  returns  to 
variable  inputs.  Ratios  greater  than  one  indicate 
decreasing  returns  while  ratios  less  than  one  indicate 
increasing  returns.  [Ref.  21:p.  88] 


In  addition  to  examining  the  problems  associated  with 
the  two  primary  surrogate  rate  measures,  Boger  and  Liao 
examined  the  problem  of  changing  cost  structures.  These 
changes  occur  as  a  result  of  changes  in  the  production  setup. 
Based  on  their  analysis,  the  researchers  concluded  that  rate 
adjustment  models  are  appropriate  only  when  applied  to  data 
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collected  from  plants  which  have  not  undergone  changes  in  cost 
structure. 

2 .   Moses 

In  1991,  Moses  [Ref.  5]  published  the  results  of  a 
study  in  which  he  investigated  and  compared  the  forecasting 
bias  for  the  learning  curve  model  and  a  rate  adjustment  model. 
Specific  objectives  of  the  study  were  as  follows:  to 
determine  if  either  the  learning  curve  model  or  rate 
adjustment  model  exhibits  consistent/systematic  bias;  to 
determine  under  what  circumstances  the  two  models  are  biased, 
and;  to  identify  the  nature  of  the  bias  (i.e.,  overestimation 
or  underestimation  of  future  costs) . 

A  simulation  approach  was  used  to  conduct  the 
research.  First,  cost  series  were  generated  under  varying 
simulated  conditions.  Then,  model  parameters  were  estimated 
by  fitting  the  learning  curve  and  rate  adjustment  models  to 
the  cost  series.  Future  costs  were  predicted  using  each  of 
the  models.  These  predicted  future  costs  were  then  compared 
with  the  actual  cost  to  measure  bias.  Finally,  the 
relationship  between  the  level  of  bias  and  the  simulated 
conditions  was  investigated  using  analysis  of  variance. 

The  simulation  was  conducted  by  varying  seven  factors 
which  had  been  found  to  affect  the  magnitude  of  model 
prediction  errors  in  prior  research.  These  seven  factors 
were: 
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1.  Data  History — the  number  of  data  points  available  to 
estimate  the  parameters  for  a  model. 

2.  Variable  Cost  Learning  Rate — the  learning  curve  exponent. 

3 .  Fixed  Cost  Burden — the  proportion  of  total  cost  comprised 
of  fixed  costs. 

4 .  Production  Rate  Trend — the  production  trend  during  the 
model  estimation  period  (i.e.,  gradual  growing  trend  or 
level  trend) . 

5.  Production  Rate  Instability/Variance — period-to-period 
fluctuations  in  the  production  rate  perhaps  caused  by 
changes  in  the  demand  for  output  or  the  supply  of  inputs, 
and  annual  budget  uncertainties. 

6.  Cost  Noise  Variance — variability  in  period-to-period 
cost — designed  to  reflect  unsystematic,  unanticipated, 
non-recurring  random  factors  (e.g.,  changes  in  the  cost, 
type  or  availability  of  input  resources,  temporary 
variations  in  the  level  of  efficiency,  and  unplanned 
changes  in  the  production  process) . 

7 .  Future  Production  Level — the  production  rate  planned  for 
the  future  relative  to  past  levels. 


Bias  was  measured  separately  for  each  model  as  follows 

PUC   -  AUC 


BIAS 


AUC 


where 


PUC  =  Predicted  unit  cost  either  the  learning  curve  or 

the  rate  adjustment  model. 
AUC  =  Actual  unit  cost  generated  by  the  cost  function. 


In  conducting  his  analysis,  Moses  found  that  the  rate 
adjustment  model  provided  unbiased  cost  estimates  while  the 
learning  curve  consistently  underestimated  actual  costs.   The 
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following  conclusions  regarding  learning  curve  bias  were 

drawn: 


1.  Learning  curve  bias  stems  from  the  fact  that  a  portion  of 
total  cost  is  fixed.  The  log  linear  relationship  between 
cost  and  quantity  assumed  by  the  learning  curve  does  not 
hold  when  fixed  costs,  which  are  not  subject  to  learning, 
are  present. 

2  .  Bias  increases  as  the  proportion  of  total  cost  made  up  of 

fixed  costs  increases.  This  relationship  holds  up  to  the 
point  where  fixed  costs  account  for  50  percent  of  total 
cost;  further  increases  above  that  level  reduce  bias. 

3  .  The  production  rate  during  the  period  of  model  estimation 

and  the  production  rate  during  the  period  for  which  costs 
are  forecast  both  affect  the  degree  of  bias.  Bias  is 
minimized  when  there  is  a  consistent  production  trend 
during  these  periods.  Bias  is  magnified  when  there  is  a 
shift  in  production  rate  trend. 

4.  The  steeper  the  learning  curve  slope,  the  greater  the 
level  of  bias.  (This  conclusion  is  based  on  the 
assumption  that  the  proportions  of  total  cost  that  are 
fixed  or  variable  remain  relatively  stable.) 

5.  The  greater  the  number  of  observations,  the  higher  degree 
of  bias. 

6.  The  further  into  the  future  predictions  are  made,  the 
greater  the  degree  of  bias. 


E.   SUMMARY 

The  review  of  previous  research  conducted  in  this  chapter, 
while  comprehensive,  is  by  no  means  all-inclusive. 
Nevertheless,  the  studies  discussed  do  provide  a  sound  basis 
for  assessing  the  current  level  of  understanding  with  respect 
to  learning  curves  and  rate  adjustment  models.   The  following 


29 


conclusions  can  be  drawn  concerning  the  research  efforts  to 

date: 


1.  The  broad  objective  of  most  of  the  studies  dealing  v/ith 
extensions  or  modifications  to  the  learning  curve  has 
been  to  investigate  the  relationship  between  production 
rate  and  cost  to  determine  whether  or  not  consideration 
of  production  rate  leads  to  improved  cost  estimates. 

2.  Findings  concerning  the  importance  of  considering  the 
effect  of  production  rate  on  unit  costs  when  predicting 
weapons  systems  costs  have  been  inconclusive.  Some 
studies  have  found  production  rate  to  be  an  important 
predictor  of  cost  variation  while  other  studies  found 
production  rate  to  be  of  little  or  no  significance. 

3.  Production  rate  affects  unit  cost  but,  in  most  cases,  not 
as  strongly  as  the  learning  (cumulative  guantity)  effect. 

4 .  The  production  rate  effect  varies  with  the  weapon  system 
and  the  specific  cost  elements. 

5.  The  influence  of  production  rate  on  unit  cost  depends  on 
the  relative  amounts  of  direct  and  indirect  costs  per 
unit  and  the  proportion  of  each  category  that  can  be 
considered  as  fixed  or  variable. 

6.  Variations  in  unit  cost  in  response  to  production  rate 
changes  stem  from  economies  of  scale.  These  variations 
are,  to  a  great  extent,  due  to  the  amortization  of  fixed 
overhead. 

7 .  Lower  unit  costs  are  associated  with  high  production 
rates  and  higher  unit  costs  are  associated  with  lower 
production  rates. 

8.  The  rate  adjustment  model,  as  described  by  Moses  (1991), 
provides  unbiased  cost  estimates.  Conversely,  the 
learning  curve  consistently  underestimates  actual  costs. 

9 .  Learning  curve  bias  stems  from  the  fact  that  a  portion  of 
total  cost  is  fixed. 

10.  Learning  curve  bias  is  affected  by  the  proportion  of 
total  cost  made  up  of  fixed  costs,  the  production  rate 
during  both  the  period  of  model  estimation  and  the  period 
for  which  costs  are  forecast,  the  slope  of  the  learning 
curve,  the  number  of  observations,  and  the  time  horizon 
for  which  predictions  are  being  made. 
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Findings  by  Moses  [Ref .  5]  concerning  the  bias  associated 
with  the  traditional  learning  curve  and  the  rate  adjustment 
model  are  noteworthy.  Nevertheless,  their  significance  is 
somewhat  tempered  by  the  use  of  simulated  data.  The  remainder 
of  this  thesis  will  focus  on  confirming  or  disconf irming  the 
results  of  the  Moses  [Ref.  5]  study  and  will  extend  the 
research  to  include  two  additional  models — a  random  walk  model 
and  a  model  incorporating  production  rate  effects. 
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III.  SAMPLE,  DATA  AND  VARIABLES 

The  purpose  of  this  chapter  is  to  provide  an  overview  of 
the  methodology  used  to  conduct  this  study.  The  chapter  will 
begin  with  a  description  of  the  population  from  which  the 
sample  was  drawn,  the  criteria  for  inclusion  of  a  weapon 
system/procurement  program  in  the  sample  and  the  specific  raw 
data  collected  for  each  weapon  system.  This  will  be  followed 
by  a  description  of  the  procedure  used  to  "expand"  the  sample. 
Next,  the  four  cost  progress  models  that  were  included  in  the 
study  will  be  introduced  along  with  the  procedure  used  for 
predicting  costs  with  these  models.  Finally,  the  procedure 
used  to  measure  bias  in  the  study  will  be  explained  and  the 
"demographic  variables"  and  the  "condition  variables"  that 
were  selected  as  likely  candidates  for  being  significant 
explainers  of  bias  will  be  defined. 

A.   SELECTION  OF  THE  SAMPLE 
1.   Data  Sources 

Data  used  in  conducting  the  study  were  obtained  from 
two  sources:  the  U.S.  Military  Aircraft  Cost  Handbook  [Ref. 
22],  and;  the  U.S.  Missile  Cost  Handbook  [Ref.  23].  Data 
contained  in  these  handbooks  were  based  on  historical  Service 
cost  data  and  reflected  the  annual  total  obligational 
authority  (TOA)   flyaway  costs  for  the  included  programs. 
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Aircraft  data  fell  into  five  major  categories:  attack 
aircraft,  fighter  aircraft,  bombers,  attack  helicopters,  and 
patrol  aircraft.  Missile  data  fell  into  four  main  categories: 
air-to-air  missiles,  air-to-surface  missiles,  surface-to-air 
missiles,  and  surface-to-surface  missiles.  All  cost  data 
contained  in  these  handbooks  was  normalized  to  a  constant 
fiscal  year  base  (FY-81) .  Conseguently ,  the  consistency  of 
this  data  was  ensured  for  all  aircraft  and  missile  programs. 

Weapons  systems  contained  in  the  aircraft  handbook 
were  U.S.  Navy,  Air  Force,  Marine  Corps  and  Army  aircraft. 
These  aircraft  were  combat-oriented  and  were  in  the  active 
U.S.  inventory  during  the  FY  1960-1980  period.  Trainers, 
reconnaissance  or  electronic  warfare  variants  and  those 
aircraft  produced  for  foreign  military  sales  were  not  included 
in  the  handbook. 

The  missile  handbook  contained  cost  information  for 
U.S.  Air  Force,  Navy,  and  Army  missile  programs  during  the 
period  FY  1961-1983. 

Costs  reflected  in  these  handbooks  were  TOA  dollars — 
the  amounts  budgeted  in  a  specific  fiscal  year.  TOA  dollars 
do  not  reflect  actual  expenditures  in  any  given  fiscal  year. 
Nevertheless,  TOA  dollars  do  provide  an  excellent  proxy  for 
actual  dollar  expenditures  because  it  is  customary  within  the 
DoD  to  ensure  that  expenditures  match  total  obligational 
authority  prior  to  the  lapsing  of  an  appropriation.   As  a 
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result,  differences  between  TOA  and  actual  expenditures  are 
normally  very  small. 

2 .  Selection  Process  and  Criteria 

Selection  of  aircraft  and  missile  programs  for 
inclusion  in  this  study  was  based  on  three  criteria:  the 
number  of  plot  points  or  fiscal  years,  the  availability  of 
airframe  cost  data  and  the  completeness  of  the  data.  Only 
programs  for  which  five  or  more  fiscal  years  worth  of  data  was 
available  were  included  in  the  study.  This  criterion  was 
required  to  ensure  the  minimum  amount  of  data  necessary  for 
statistically  fitting  the  cost  progress  models.  The  latter 
two  criteria  were  established  to  ensure  that  meaningful 
analyses  could  be  conducted.  As  a  result,  only  aircraft 
programs  which  had  complete  quantity  and  airframe  cost  data 
were  included  in  the  study.  In  addition,  only  missile 
programs  for  which  complete  quantity,  and  guidance  and 
control/airframe  costs  were  available  were  included  in  the 
study. 

3 .  Programs  Selected  for  Study 

Based  on  the  three  criteria  described  above,  forty-six 
weapons  procurement  programs  (fourteen  missile  programs  and 
thirty-two  aircraft  programs)  were  selected  for  inclusion  in 
this  study.  The  specific  aircraft  and  missile  programs  that 
comprised  the  sample  are  included  in  Tables  1  and  2 
respectively. 
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TABLE  1 

AIRCRAFT  PROCUREMENT  PROGRAMS  INCLUDED  IN  THE  STUDY 

1.  A-3A/B  12.  AH- IS  23.  F-16A 

2.  A-4C  13.  AV-8A  24.  F-101A/B/C 

3.  A-4E  14.  B-47B/E  25.  F-102A 

4.  A-4A/B  15.  F-3A/B/C  26.  F-105D 

5.  A-4E/F  16.  F-4A  27.  F-105B/D 

6.  A-6A  17.  F-4B  28.  F-111F 

7.  A-6E  18.  F-4E  29.  P-2H 

8.  A-7D  19.  F-4J  30.  P-3A 

9.  A-7E  20.  F-4A/B  31.  P-3C 

10.  A-37B  21.  F-8D/E  32.  S-3A 

11.  AH-1G  22.  F-14A 
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TABLE  2 


MISSILE  PROCUREMENT  PROGRAMS  INCLUDED  IN  THE  STUDY 


1.   PHOENIX  (AIM-54A) 


8.   STANDARD  ER  (RIM-67B/ 
B-l/C-1) 


2.  SIDEWINDER  (AIM-9D/G)     9.   STANDARD  ER  (RIM-67A) 

3.  SIDEWINDER  (AIM-9H)       10.   STANDARD  MR  (RIM-66A) 

4.  SIDEWINDER  (AIM-9H/L)    11.   STANDARD  MR  (RIM-66B) 


5.   SPARROW  (AIM-7E) 


12.   TALOS  (RIM-8E) 


6.   SPARROW  (AIM-7F) 


13.   TARTAR  (RIM-24B) 


7.   SHRIKE  (AGM-45A) 


14.   TERRIER  (RIM-2E) 


TABLE  2 
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B.   COMPILATION  OF  PROGRAM  DATA 

Once  the  sample  of  aircraft  and  missile  procurement 
programs  had  been  selected,  the  following  data  were  obtained 
for  each  program  in  the  sample: 

1.  Program  Name — name  of  the  weapon  system. 

2.  Manufacturer — name  of  the  prime  contractor. 

3.  Military  Service — identified  which  Service  branch (es) 
procured  the  weapon  system. 

4.  Program  Type — identified  the  nature  of  each  weapon 
system,  i.e.,  aircraft  or  missile. 

5.  Mission — identified  each  weapon  system  according  to  its 
primary  mission.  Aircraft  program  types  were: 
fighter/attack,  fighter,  attack,  bomber,  patrol,  and  attack 
helicopter.  Missile  program  types  were:  air-to-air,  air-to- 
surface,  surface-to-air,  and  surface-to-surface. 

6.  Modification — identified  whether  a  particular  weapon 
system  was  an  entirely  new  design  or  a  modification  of  an 
existing  design. 

7.  Combined  Program — identified  whether  a  procurement 
program  included  several  versions  of  a  given  weapon  system 
or  only  one  version. 

8.  Fiscal  Year — identified  the  fiscal  years  during  which 
quantities  of  a  particular  weapon  system  were  procured. 

9.  Quantity — identified  the  number  of  units  of  each  weapon 
system  procured  in  a  given  fiscal  year. 

10.  Aircraft  Airframe  Cost/Missile  Guidance  and  Control 
Airframe  Cost 
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C.   EXPANSION  OF  THE  SAMPLE 

Following  the  compilation  of  data  described  in  the 
previous  section,  the  original  data  sample  of  forty-six 
programs  was  partitioned  into  121  cost  series.  This 
"expansion  of  the  sample"  was  accomplished  by  dividing  each 
program  cost  series  into  individual,  consecutive  year-to-date 
cost  series.  For  example,  if  a  particular  procurement  program 
had  cost  data  available  for  seven  fiscal  years,  e.g.,  FY  1970- 
1976,  this  single  cost  series  could  be  expanded  into  four 
separate  cost  series  as  shown  below. 

Cost  Series  #  1:  FY  1970-1973  (used  to  predict  1974  cost) 

Cost  Series  #2:  FY  1970-1974  (used  to  predict  1975  cost) 

Cost  Series  #3:  FY  1970-1975  (used  to  predict  1976  cost) 

Cost  Series  #4:  FY  1970-1976  (used  to  predict  1977  cost) 

The  initial  cost  series  for  each  weapon  system  included  in  the 
sample  was  comprised  of  data  from  the  first  four  fiscal  years 
of  that  particular  program  (the  minimum  number  of  years  needed 
to  estimate  the  cost  models)  .  Each  subsequent  cost  series  for 
a  given  program  was  then  created  by  additionally  including  the 
data  point  for  the  next  fiscal  year  in  the  existing  cost 
series.  By  partitioning  the  sample  size  in  the  study  in  this 
manner,  it  was  possible  to  simulate  actual  usage  of  the 
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various  cost  progress  models  over  time  and  evaluate  their 
performance  under  varying  data  availability  conditions. 

D.   SELECTION  OF  COST  PROGRESS  MODELS 

Previous  research  has  demonstrated  that  three  factors  are 
particularly  useful  in  predicting  the  future  costs  of  weapons 
systems:  past  cost,  cumulative  quantity,  and  production  rate. 
Numerous  models  have  been  introduced  in  an  effort  to  improve 
the  quality  of  cost  estimates  over  those  obtained  using  the 
traditional  learning  curve  model.  These  models  are 
differentiated  by  the  explanatory  variables  included  and  the 
underlying  assumptions  with  regard  to  the  relative  importance 
of  past  cost,  cumulative  quantity  and  production  rate  in 
predicting  future  costs. 

In  addition,  these  models  may  be  differentiated  by  the 
period  or  length  of  time  over  which  data  is  observed  and  used 
in  creating  a  forecast.  Some  models  assume  that  future  cost 
depends  only  on  the  most  recent  cost,  quantity  and/or 
production  rate  levels.  However,  other  models  assume  that 
cost,  quantity  or  production  rate  levels  from  early  in  a 
program's  life  are  also  significant  and,  as  a  result, 
specifically  consider  data  covering  the  full  production  life. 
Regardless  of  which  approach  is  used,  there  are  both 
advantages  and  disadvantages  associated  with  the  use  of  data 
comprised  of  only  recent  observations  or  data  comprised  of 
both  recent  and  earlier  observations. 
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Use  of  recent  observations  may  increase  the  relevance  of  model 
results  in  light  of  the  current  situation  but  sacrifices  any 
information  reflected  in  earlier,  historical  data.  In 
addition,  models  based  only  on  a  few  recent  observations  are 
more  susceptible  to  random  noise.  Conversely,  use  of  older 
observations  may  reduce  the  relevance  of  results  or  lead  to 
results  that  are  not  representative  of  the  current  situation. 
However,  the  use  of  additional  data  reduces  the  impact  of 
random  variance  in  recent  observations. 

Finally,  in  addition  to  the  two  factors  discussed  above, 
cost  progress  models  may  be  differentiated  by  the  form  of  the 
assumed  relationship  between  the  dependant  variable — cost — and 
the  potential  explanatory  variables — past  cost,  cumulative 
quantity,  and  production  rate.  Relationships  may  be  linear, 
log  linear  or  some  other  form. 

Four  alternative  models  were  selected  for  inclusion  in 
this  study:   a  random  walk  model,  the  traditional  learning 
curve  model,  a  model  which  expresses  unit  cost  as  a  function 
of  past  cost  and  production  rate  (the  fixed-variable  model) 
and  the  common  rate  adjustment  model. 

1.   Random  Walk  Model 

The  random  walk  model  assumes  that  future  cost  is  a 
function  of  past  cost;  however,  only  the  most  recent  cost  is 
relevant.  Any  deviation  from  predicted  cost  is  considered 
random  deviation. 
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The  model  is  expressed  as  follows: 


where 

UCt  =  Predicted  unit  cost  in  time  period  t, 
UCt  _  1  =  Actual  unit  cost  in  time  period  t  -  1 
t  =  Time  period. 


This  model  was  selected  for  inclusion  in  the  study  for  two 
reasons.  First,  it  exemplifies  the  cost  estimation  method 
used  by  budget  programmers  when  there  is  only  very  limited 
historical  data.  Under  these  circumstances,  future  cost 
projections  are  often  based  on  actual  costs  in  the  previous 
period.  Second,  the  random  walk  model  is  the  most  basic  and 
naive  cost  estimation  model  and,  as  such,  provides  a  useful 
benchmark  for  evaluating  the  performance  of  other  more 
sophisticated  models. 

2.   Traditional  Learning  Curve  Model 

The  second  model  selected  for  inclusion  in  the  study 
was  the  traditional  learning  curve.  This  model  assumes  that 
future  cost  is  a  function  of  both  past  cost  and  cumulative 
quantity  produced.  Moreover,  all  historical  cost  data  is 
considered  to  be  relevant. 
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The  traditional  learning  curve  model  assumes  a  log  linear 
relationship  between  cost  and  cumulative  quantity  and  is 
expressed  as  follows: 

UCt  =  aQ? 


where 


UCt  =  Incremental  unit  cost  of  item  at  quantity  Q. 
Qt  =  Cumulative  quantity  produced  as  of  period  t, 
a      =  Theoretical  first  unit  cost. 

b      =      °^  r  =  Learning  curve  exponent . 
r   =  Learning  rate. 


The   traditional   learning   curve   model   was   selected   for 
inclusion   in  the   study  because   it   is   the  most  widely 
researched  cost  progress  model  and  represents  the  foundation 
on  which  other  cost  progress  model  variants  are  based. 
3.   Fixed-Variable  Model 

The  third  model  selected  for  inclusion  in  the  study 
was  the  fixed-variable  model.  This  model  specifically 
addresses  the  relationship  between  total  unit  cost  and  unit 
variable  costs  and  unit  fixed  cost.  Variable  cost  per  unit 
remains  constant;  however,  fixed  cost  per  unit  varies 
depending  on  the  production  volume  because  total  fixed  cost  is 
allocated  by  spreading  it  over  the  total  volume  of  output. 
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The  fixed-variable  model  is  expressed  as  follows 


UC.  =  a   + 


PR. 


where 


UCt  =  Unit  cost  in  period  t. 

a  =  Variable  cost  per  unit. 

b  =  Standard  fixed  cost  per  unit. 

FRt  =  Production  rate  =  Qt  +  QAVG. 

Qt  =  Production  quantity  in  period  t. 

QAVG  =  Average  production  quantity  per  period. 


This  model  was  included  in  the  study  because  it  explicitly 
considers  the  impact  of  production  rate  on  unit  cost  through 
the  allocation  of  fixed  overhead.  In  contrast  to  the 
traditional  learning  curve  model  (which  includes  cumulative 
quantity  but  not  production  rate  as  an  explanatory  variable) , 
the  fixed-variable  model  includes  production  rate  but  ignores 
cumulative  quantity  as  an  explanatory  variable. 
4.   Bemis  Production  Rate  Adjustment  Model 

The  final  model  included  in  the  study  is  the  most 
widely  used  rate  adjustment  model.  This  model,  popularized  by 
Bemis  [Ref.  16],  was  developed  by  augmenting  the  traditional 
learning  curve  model  with  a  production  rate  term. 
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The  model  is  expressed  as  follows: 

UCt  =  aQc   R< 

where 


UCZ   =  Predicted  unit  cost  at  quantity  Q  and  production  rate 

per  period  R. 
Qt     =  Cumulative  quantity  produced  as  of  period  t. 
Rt     =  Production  rate  in  period  t. 
a      =  Theoretical  first  unit  cost. 
b      =  Learning  curve  exponent . 
c      =  Production  rate  exponent. 


The  Bemis  production  rate  adjustment  model  was  included  in  the 
study  because  it  considers  both  cumulative  quantity  and 
production  rate  (in  addition  to  past  cost).  Hence,  it  is  the 
most  comprehensive  of  the  four  models  in  the  study. 


E.   ANALYSIS  OF  BIAS 

1.   Unit  Cost  Prediction 

In  order  to  assess  the  bias  exhibited  by  the  random 
walk  model,  the  traditional  learning  curve  model,  the  fixed- 
variable  model,  and  the  Bemis  model,  predicted  unit  costs  were 
estimated  by  applying  each  of  the  four  models,  in  turn,  to  the 
actual  cost  series.  The  following  paragraphs  describe  how 
this  procedure  was  accomplished. 

The  random  walk  model  assumes  that  unit  cost  in  the 
next  period  is  the  same  as  unit  cost  in  the  current  period. 
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Consequently,  the  predicted  unit  cost  for  any  period  (t)  was 
simply  the  actual  cost  from  the  preceding  period  (t-1) . 

Predicted  unit  costs  for  the  remaining  three  models 
were  derived  in  the  following  manner.  First,  each  of  the 
models  was  separately  fit  to  the  initial  cost  series 
(comprised  of  the  first  four  fiscal  years1  data)  for  each 
weapon  system  to  derive  the  models'  parameters.  Then,  the 
appropriate  data  values  for  cumulative  quantity  and/or 
production  rate  from  period  five  were  input  into  each  of  the 
models  to  obtain  estimated  unit  costs  for  period  five.  These 
two  steps  were  then  repeated  for  each  remaining  cost  series 
for  the  various  weapons  systems  until  predicted  costs  had  been 
derived  for  every  fiscal  year  for  which  actual  cost  data  was 
available  (e.g.,  models  were  next  estimated  on  five  years 
worth  of  data,  then  used  to  predict  the  cost  for  year  six) . 
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2.   Bias  Measurement-The  Dependant  Variable 

Once  predicted  unit  costs  had  been  computed  for  each 
cost  series  with  each  model,  a  measure  of  bias  was  determined 
for  each  prediction  as  follows: 

BIAS  =     PUC  ~  AUC 
AUC 


where 


BIAS  =  Percentage  difference  between  predicted  unit  cost 

and  actual  unit  cost. 
PUC     =  Predicted  unit  cost  from  the  particular  model  of 

interest . 
AUC     =  Actual  unit  cost  obtained  from  original  unit  cost 

data. 


Positive  BIAS  values  indicate  a  model  has  overestimated  actual 
future  cost;  negative  BIAS  values  indicate  a  model  has 
underestimated  actual  future  cost.  BIAS  values  of  zero 
indicate  the  predicted  cost  and  actual  future  cost  are 
identical  and  the  associated  model  is  unbiased. 

As  in  the  earlier  study  conducted  by  Moses  [Ref.  5], 
BIAS  represented  the  dependent  variable  in  the  statistical 
analysis.  The  basic  objective  of  the  study  was  to  determine 
what  factors  or  conditions  are  useful  in  explaining  variance 
in  BIAS. 


46 


The  measures  of  bias  and  associated  labels  for  each  of  the 
four  models  were  as  follows: 

Bias  for  the  Random  Walk  Model:  BIASRW 

Bias  for  the  Learning  Curve  Model:  BIASLC 

Bias  for  the  Fixed-Variable  Model:  BIASFV 

Bias  for  the  Bemis  Model:  BIASBE 

Given  121  cost  series  (from  the  46  procurement  programs) , 
there  were  121  separate  measures  of  bias  for  each  model. 
3.   Explanatory  Variable  Selection 

Model  performance  in  prediction  (i.e.,  the  degree  and 
direction  of  bias)  depends  on  the  circumstances  in  which  the 
model  is  used.  Two  broad  categories  of  factors  which  might 
influence  model  performance  and,  hence,  be  useful  in 
explaining  bias,  were  identified  in  the  study.  The  first 
category  consisted  of  "demographic  variables"  and  the  second 
category  consisted  of  "condition  variables".  Together,  these 
two  groups  of  variables  were  the  independent  variables  in  the 
statistical  analysis.  Table  3  summarizes  the  independent 
variables  and  their  corresponding  labels. 
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TABLE  3 


INDEPENDENT  VARIABLES 


Demographic  Variables 


Program  Type 


Modification 


Military  Service 


Mission 


Condition  Variables 


Burden 


Cost  Variance 


Learning  Rate 


Production  Rate  Variance 


Future  Production  Level 


Past  Production  Trend 


Plot  Points 


Labels 
TYPE 
MOD 


MILSERV 
MISSION 


BURDEN 

CVAR 

LRATE 

RATEVAR 

FUTUPROD 

BEGTREND,  ENDTREND 

PLOTPNTS 
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a.  Demographic  Variables 

The  demographic  variables  describe  characteristics 
of  the  weapons  procurement  program.  Four  demographic 
variables  were  considered  in  the  study: 

1 .  Program  Type 

2 .  Modification 

3.  Military  Service 

4.  Mission 

These  particular  demographic  variables  were 
selected  for  investigation  because  they  represent  readily 
apparent  characteristics  of  the  various  programs  which  may 
affect  model  bias. 

b.  Condition  Variables 

In  addition  to  the  four  demographic  variables, 
seven  condition  variables  were  considered  in  the  study: 

1.  Burden 

2.  Cost  Variance 

3 .  Learning  Rate 

4 .  Production  Rate  Variance 

5.  Future  Production  Level 

6.  Past  Production  Trend 

7.  Plot  Points 
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The  label  "condition  variables"  is  used  here  because  each  of 
these  variables  in  some  manner  indicates  something  about 
conditions  that  existed  during  the  weapon  system  procurement. 
These  are  analogous  independent  variables  (factors)  to  those 
examined  by  Moses  [Ref.  5].  The  following  paragraphs  describe 
each  of  these  variables  along  with  the  underlying  rationale 
for  their  inclusion  in  the  study. 
(1)    Burden 

Burden  (BURDEN)  indicates  the  percentage  of 
total  unit  cost  made  up  of  fixed  cost.  Burden  is  measured  as 
follows: 

BURDEN  = 


a  +  b 


where 


a  =  Variable  cost  per  unit   {constant)  . 
b  =  Standard  fixed  cost  per  unit. 

Note:     a  and  b  were  estimated  parameters  from  the  fixed- 
variable  model. 


Burden  was  included  in  the  study  because  past  research  (Linder 
and  Wilbourn  [Ref.  10],  Moses  [Ref.  5]  and  Smunt  [Ref.  24]) 
has  shown  that  burden  directly  influences  the  impact  of 
changes  in  production  rate  on  unit  cost.  As  production  rate 
increases,  the  cumulative  quantity  produced  during  a  period 
increases  and  the  variable  cost  per  unit  decreases  due  to  the 
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incidence  of  learning.  In  addition,  as  production  rate  in  a 
period  increases,  fixed  cost  per  unit  is  reduced  as  total 
fixed  cost  is  spread  over  a  larger  production  output.  The 
impact  of  production  rate  increases  as  the  proportion  of  total 
cost  made  up  of  fixed  costs  increases.  As  a  result,  the 
relative  bias  of  the  various  models  may  depend  on  the  level  of 
burden. 

(2)    Cost   Variance 

Cost  variance  (CVAR)  indicates  the  amount  of 
unsystematic  variation  in  unit  cost  that  may  result  from 
unanticipated,  non-recurring,  random  factors.  Examples 
include  changes  in  the  cost,  type  or  availability  of  input 
resources,  temporary  fluctuations  in  efficiency,  and  unplanned 
changes  in  the  production  process. 

By  assessing  the  amount  of  unsystematic 
variation  associated  with  various  weapons  procurement 
programs,  it  may  be  possible  to  determine  whether  there  is  a 
relationship  between  cost  stability  and  bias.  For  this 
reason,  cost  variance  was  examined  in  the  study. 
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Cost  variance  was  measured  as  follows 


zl     \c~    ~    CAVG\ 


•VAR  =    &ZH 


n 


where 


CVAR  =  Cost  variance. 

Ct        =  Unit  cost  for  a  given  production  period. 

CAVG    -  Average  unit  cost  for  all  production  periods  to  date, 

n  =  Total  number  of  production  periods . 


(3)    Learning  Rate 

Learning  rate  (LRATE)  measures  the  decrease  in 
per  unit  cost  that  occurs  as  the  quantity  of  units 
manufactured  doubles.  It  is  affected  by  the  type  of 
production  process  and  the  complexity  of  the  product  design. 
Smunt  [Ref.  24]  found  that  the  degree  of  learning  in  the 
underlying  production  process  determines  the  improvement  in 
prediction  accuracy  that  results  from  including  a  learning 
parameter  in  a  model.  Learning  rate  was  examined  in  the  study 
to  determine  the  nature  of  the  relationship  between  learning 
rate  and  bias  for  the  various  models. 
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Learning  rate  was  measured  as  follows 


mere 


=  2' 


i  =  Learning  rate  expressed  as  a  percentage 
b  =  Learning  curve  exponent:  as  estimated 

by  the  traditional  learning  curve  model 


(4)    Production  Rate  Variance 

Production  rate  variance  (RATEVAR)  reflects  the 
severity  of  period-to-period  fluctuations  in  production  rate. 
These  fluctuations  may  result  from  either  changes  in  demand 
for  a  particular  weapon  system  or  changes  in  the  cost  or 
availability  of  production  inputs.  Production  rate  variance 
was  measured  as  follows: 


E  \Qt  -  Qavg\ 

RATEVAR  =   22 


n 


where 


RATEVAR  =  Rate  variance. 

Qt  =  Quantity  of  units  produced  in  the  current  period 

@avg  =  Average  quantity  of  units  produced  for  ail 

periods  to  date. 
n  =  Total  number  of  periods  to  date. 
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(5)    Future   Production   Level 

Future  production  level  (FUTUPROD)  indicates 
whether  the  level  of  production  in  the  next  period  (the  period 
for  which  cost  is  to  be  forecast  using  the  model)  is  high  or 
low  relative  to  the  current  period.  Future  unit  costs  are 
predicted  using  cost  progress  models  fit  to  past  production 
data.  As  a  result,  the  accuracy  of  the  various  models  may  be 
affected  if  the  production  level  in  the  period  for  which  costs 
are  being  estimated  varies  significantly  from  the  production 
levels  that  existed  during  previous  periods.  Future 
production  level  was  examined  in  the  study  to  determine 
whether  production  growth  and  production  cutbacks  affect  the 
tendencies  of  the  various  models  to  over/underestimate  unit 
costs. 

Future   production   level   was   measured   as 


follows 


FUTUPROD  =   log  (—?] 

Ql 


'.•/here 


FUTUPROD  =  Future  production  level. 

QN  =  Production  level  for  the  next  period  for  which 

costs  are  being  forecast . 
QL  =  Production  level  for  the  last    (most  recent) 

period. 
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(6)    Past   Production   Trend 

Past  production  trend  indicates  the  pattern  of 
production  volume  associated  with  each  weapon  system  included 
in  the  study.  Two  variables,  3EGTREND  and  ENDTREND,  were  used 
to  reflect  how  production  volume  was  changing  at  the  beginning 
and  end  of  each  cost  series  to  which  models  were  fit. 
Production  rate  per  period  may  be  low  initially  in  order  to 
work  out  bugs  and  ensure  a  stable  product  design  prior  to  full 
scale  production.  Alternatively,  initial  production  rate  per 
period  may  be  high  if  the  current  weapon  system  represents  an 
updated  version  of  an  already  existing  system  with  only  minor 
modifications . 

Production  rate  per  period  may  also  vary  at  the 
end  of  the  production  run  depending  on  whether  the  program  is 
abruptly  cancelled,  gradually  phased  out  or  continued  at  some 
minimum  level  in  order  to  maintain  a  warm  production  base. 
These  two  production  trend  variables  were  included  in  the 
study  to  determine  whether  the  bias  associated  with  the 
various  models  was  related  to  past  production  trend. 
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The  production  trend  variables  were  measured  as  follows 

3EGTREND  =     ~AVG  ~   °l 
Oavg 

and 

^L  ~  *AVG 


ENDTREND   = 


AVG 


where 


3EGTREND  =  Production  trend  at  the  beginning  of 

the  production  run. 
ENDTREND  =  Production  trend  for  the  most  recent  production 

period. 
Qx  =  Quantity  produced  in  first  production  period. 

QL  =  Quantity  produced  in  most  recent  period. 

Qavg  =  Average  quantity  per  period  produced  through 

current  period. 


Positive  values  for  BEGTREND  and  ENDTREND  indicate  production 
trends  that  are  increasing  in  volume. 
(7)    Plot   Points 

The  final  independent  variable  included  in  the 
study  was  the  number  of  plot  points  (PLOTPNTS) .  Plot  points 
indicates  the  number  of  data  points  available  to  estimate 
model  parameters.  The  accuracy  of  the  learning  curve, 
fixed-variable,  and  Bemis  models  should  improve  as  the  amount 
of  data  available  during  the  model  estimation  period 
increases.   Nevertheless,  if  a  model  is  inherently  biased, 
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increasing  the  number  of  plot  points  will  not  necessarily 
eliminate  the  bias  associated  with  the  model.  The  number  of 
plot  points  was  considered  in  the  study  to  determine  whether 
or  not  the  bias  associated  with  the  various  models  could  be 
explained  in  terms  of  the  availability  of  past  production 
data . 

F.   SUMMARY 

This  chapter  has  provided  an  overview  of  the  sample  and 
variables  to  be  used  to  investigate  bias.  The  discussion 
began  with  a  description  of  the  sample  selection  process,  and 
a  summary  of  the  data  collected  for  each  program  included  in 
the  sample.  This  was  followed  by  a  description  of  the 
procedure  used  to  expand  the  sample  from  46  observations 
(programs)  to  121  cost  series.  Next,  the  four  cost  progress 
models  included  in  the  study — the  random  walk  model,  the 
traditional  learning  curve  model,  the  fixed-variable  model, 
and  the  Bemis  production  rate  adjustment  model  were  described 
along  with  the  rationale  for  their  selection.  The  cost 
estimation  procedure  was  then  explained  and  a  method  for 
measuring  bias  was  introduced.  Finally,  two  categories  of 
independent  variables — demographic  variables  and  condition 
variables—were  introduced  to  be  evaluated  as  potential 
sources  of  model  bias. 
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IV.   ANALYSIS  AND  FINDINGS 

Chapter  III  introduced  eleven  variables — four  demographic 
variables  and  seven  condition  variables — which  might  influence 
the  performance  of  alternative  cost  progress  models  and, 
hence,  be  useful  in  explaining  the  cost  prediction  bias  of 
these  models.  This  chapter  describes  the  statistical 
procedures  that  were  performed  to  assess  the  significance  of 
each  of  these  variables  in  explaining  bias.  It  will  begin  by 
providing  an  overview  of  the  statistical  tests  that  were  used 
to  analyze  the  variables.  This  will  be  followed  by  a 
presentation  of  some  general  findings  with  respect  to  the 
performance  of  the  four  alternative  cost  progress  models. 
Next,  model  specific  findings  regarding  the  significance  of 
each  of  the  eleven  explanatory  variables  will  be  presented. 
Finally,  the  results  of  the  analysis  will  be  summarized  and 
conclusions  will  be  presented. 

A.   STATISTICAL  PROCEDURES 

Prior  to  conducting  statistical  analyses,  the  distribution 
of  values  for  each  of  the  dependent  and  independent  variables 
was  assessed.  The  presence  of  extreme  values  within  the  data 
set  could  unduly  influence  the  outcomes  of  the  statistical 
analyses.  Accordingly,  variable  values  which  lay  beyond  three 
standard  deviations  from  the  mean  for  the  variable  were 
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truncated.  Truncation  involved  replacing  extreme  values  v/ith 
values  equal  to  three  times  the  standard  deviation  for  the 
appropriate  variable.  This  approach  was  applied  because  it 
reduces  the  influence  of  outliers  on  the  results  of  the 
statistical  analyses  without  discarding  and,  consequently, 
ignoring  the  impact  of  these  observations. 

Once  all  extreme  values  had  been  identified  and  truncated, 
the  dependency  of  BIAS  on  the  demographic  variables  was 
evaluated  for  each  model  using  analysis  of  variance  (ANOVA) . 
After  determining  the  significance  of  the  demographic 
variables  in  explaining  BIAS,  the  significance  of  the  seven 
condition  variables  and  MOD  was  assessed  using  both  simple  and 
multiple  linear  regression  analysis.  Finally,  three  different 
sets  of  Spearman  and  Pearson  correlation  analyses  were 
conducted.  First,  correlation  coefficients  were  computed  for 
the  measures  of  BIAS  from  the  four  different  models.  Next, 
correlations  between  the  condition  variables  (including  MOD) 
and  BIAS  were  examined  for  each  model.  Then  values  for  the 
condition  variables  (including  MOD)  were  correlated  with  each 
other  in  an  effort  to  detect  potential  multicollinearity . 

In  conducting  tests  of  statistical  significance,  findings 
v/ith  alpha  values  less  than  0.01  were  considered  significant. 
When  analyzing  pairwise  correlations  between  the  explanatory 
variables,  correlation  coefficients  larger  than  0.50  were 
regarded  as  offering  strong  evidence  that  multicollinearity 
might  be  a  problem. 
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B.   ANOVA  RESULTS 

Analysis  of  variance  was  performed  for  each  of  the  four 
cost  progress  models  to  determine  whether  or  not  prediction 
bias  could  be  explained  by  any  of  the  demographic  variables. 
The  following  demographic  variables  were  examined: 

1.  TYPE 

2.  MILSERV 

3.  MISSION 

4.  MOD 

5.  TYPE  x  MOD  (interaction  variable) 

None  of  the  demographic  variables  tested  were  significant  in 
explaining  the  bias  of  cost  predictions  made  with  the  random 
walk,  fixed-variable  or  Bemis  models.  However,  MISSION,  MOD 
and  the  TYPE  x  MOD  interaction  variable  were  all  found  to  be 
significant  in  their  ability  to  explain  variations  in  the  bias 
of  learning  curve  cost  predictions.  ANOVA  results  for  the 
learning  curve  model  are  provided  in  Table  4.  Findings 
concerning  the  significance  of  MOD  in  explaining  variations  in 
the  bias  of  learning  curve  cost  predictions  will  be  discussed 
later,  along  with  the  regression  results.  Differences  in 
learning  curve  bias  due  to  MISSION  and  TYPE  are  simply  noted. 
No  hypothesis  was  offered  to  expect  differences  in  learning 
curve  cost  prediction  bias  in  relation  to  MISSION  and  TYPE. 
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Consequently,  findings  related  to  these  factors  were  not 
pursued  further  but  are  mentioned  here  for  possible 
investigation  in  future  research.  Based  on  the  fact  that  the 
utility  of  these  variables  in  explaining  model  bias  was 
limited  to  the  learning  curve  model,  the  remainder  of  the 
study  was  devoted  to  examining  the  significance  of  the 
condition  variables  (including  MOD)  in  explaining  model  bias. 

C.   OVERALL  MODEL  PERFORMANCE 
1.   Mean  Bias 

Summary  results  for  the  average  level  of  bias 
associated  with  cost  predictions  made  using  the  random  walk, 
learning  curve,  fixed-variable  and  Bemis  models  are  provided 
in  Table  5.  The  mean  bias  of  predicted  unit  costs  estimated 
with  the  four  models  ranged  from  -0.008313  for  the  learning 
curve  to  0.375045  for  the  fixed-variable  model.  The  results 
show  that  on  average,  the  learning  curve  underestimated  unit 
costs  by  approximately  0.33%  while  the  Bemis,  random  walk  and 
fixed-variable  models  overestimated  unit  costs  by 
approximately  2.9%,  4.6%  and  37.5%  respectively.  Bias 
measures  for  all  four  models  were  skewed  in  the  positive 
direction.  Hence,  measures  of  the  median  bias  were  examined 
in  an  effort  to  obtain  a  more  objective  assessment  of  model 
performance. 
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TABLE  5 


RANK  ORDERING  OF  MODELS  BY  MAGNITUDE 
OF  POSITIVE  BIAS  (Lowest  10  Highest) 


Model 

Mean  Bias 

Median 
Bias 

Standard 
Deviation 

Skewness 

Learning 
Curve 

-0.008313 

-0.060802 

0.385325 

3  .708457 

Bemis 

0.029472 

-0.012925 

0.  339456 

0.838916 

Random 

Walk 

0.045842 

0.015789 

0.199953 

1.396772 

Fixed- 
Variable 

0.375045 

0.188371 

0.929000 

3.391860 
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2.   Median  Bias 

The  median  bias  of  predicted  unit  costs  estimated  with 
the  four  models  ranged  from  -0.060802  for  the  learning  curve 
to  0.188371  for  the  fixed-variable  model.  When  the  median 
bias  measurements  for  the  various  models  were  compared,  the 
relative  ordering  of  the  models  according  to  magnitude  of 
positive  bias  remained  the  same.  However,  the  magnitude  of 
positive  bias  decreased  significantly  in  all  cases.  This 
suggests  that  some  observations  had  a  very  large  positive 
bias,  causing  the  mean  bias  to  be  more  positive  than  the 
median.  The  direction  of  bias  remained  unchanged  for  all 
models  except  for  the  Bemis  model.  In  this  case,  mean  model 
bias  was  approximately  2.9%  while  median  model  bias  was 
approximately  -1.3%.  These  results  seemed  to  indicate  that 
the  learning  curve  and  Bemis  models  underestimate  unit  cost 
(provide  low  unit  cost  estimates)  while  the  random  walk  model 
and  especially  the  fixed-variable  model  overestimate  unit  cost 
(provide  high  unit  cost  estimates)  .  Perhaps  not 
coincidentally,  both  the  learning  curve  model  and  the  Bemis 
model  use  cumulative  quantity  to  predict  future  cost,  while 
neither  the  random  walk  model  nor  the  fixed-variable  model 
contain  a  cumulative  quantity  term. 
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3.   Model  Bias  Correlations 

Both  Pearson  correlation  coefficients  and  Spearman 
correlation  coefficients  were  computed  for  zhe  values  of  bias 
associated  with  each  of  the  four  cost  progress  models.  The 
results  of  these  correlation  analyses  are  provided  in  Table  6. 
All  Spearman  correlation  coefficients  were  positive  and 
significant;  correlation  values  ranged  from  0.27220  to 
0.58048.  However,  these  results  were  not  completely  confirmed 
by  the  computed  Pearson  correlation  coefficients.  Values 
obtained  from  the  Pearson  correlation  analysis  indicated 
significant  bias  correlations  for  only  three  of  the  six  paired 
model  combinations:  BIASRW-BIASLC,  BIASRW-BIASBE,  and  BIASBE- 
BIASFV.  Significant  Pearson  correlation  coefficients  ranged 
from  0.26401  to  .063335.  Both  correlation  analyses  indicated 
that  the  strongest  positive  correlation  existed  between  random 
walk  model  bias  and  learning  curve  model  bias.  The  second 
highest  correlation  in  both  analyses  existed  between  random 
walk  model  bias  and  Bemis  model  bias.  The  third  relationship 
which  was  correlated  and  significant  in  both  analyses  was 
Bemis  model  bias  and  fixed-variable  model  bias. 

Based  on  the  outcomes  from  the  correlation  analyses, 
it  was  concluded  that  there  is  evidence  of  a  positive 
correlation  between  bias  for  all  paired  model  combinations. 


65 


Table  6 


CORRELATION  ANALYSES  OF  MODEL  PREDICTION  BIAS 


Spearman  Correlation  Coefficients 


BIASRW 

BIASLC 

BIASFV 

BIASBE 

BIASRW 

1.00000 
0.0 

0.58048* 
0.0001 

0.27220* 
0.0025 

0.34698* 
0.0001 

BIASLC 

0. 63335* 
0.0001 

1.00000 
0.0 

0.35518* 
0.0001 

0.43751* 
0.0001 

BIASFV 

-0.02073 
0.8215 

-0.04602 
0.6162 

1.00000 
0.  0 

0.54488* 
0. 0001 

BIASBE 

0.26401* 
0.0034 

0.13296 
0.1460 

0.35905* 
0.0001 

1.00000 
0.  0 

Pearson  Correlation  Coefficients 


Correlation  Coefficients/ Prob>\R\  under  H0:   Rho=0 
*  Indicates  a   <.  0.01 
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That  being  the  case,  it  seemed  reasonable  no  suspect  that,  to 
the  extent  prediction  bias  was  positively  correlated  among  the 
various  models,  the  random  walk,  learning  curve,  fixed- 
variable  and  Bemis  models  might  perform  in  a  similar  manner 
under  the  same  circumstances.  Moreover,  it  was  possible  that 
the  significance  of  various  explanatory  variables  in 
explaining  prediction  bias  might  be  similar  between  models 
whose  prediction  bias  was  highly  correlated.  In  order  to 
resolve  these  issues,  correlation  analyses  and  regression 
analyses  were  used  to  study  the  relationship  between  model 
bias  and  the  explanatory  variables.  The  following  paragraphs 
provide  the  results  of  these  analyses  for  each  of  the  four 
models. 

D.   REGRESSION  ANALYSES  AND  RESULTS 

The  significance  of  the  seven  condition  variables  and  one 
demographic  variable — MOD — was  assessed  using  regression 
analysis.  First,  simple  regression  analyses  were  performed  to 
independently  test  the  significance  of  these  independent 
variables  in  explaining  prediction  bias  when  considered  in 
isolation.  Then  multiple  regression  analyses  were  conducted 
to  determine  the  significance  of  these  same  independent 
variables  in  explaining  prediction  bias  while  controlling  for 
the  effects  of  the  other  independent  variables.  Pairwise 
correlations  were  additionally  determined  among  both 
independent  and  dependent  variables.   The  following  sections 
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describe  the  results  of  these  analyses  for  each  of  the  four 
models.  The  discussion  of  the  results  will  be  organized  by 
model  type  in  the  following  sections.  Summary  tables 
containing  average  bias  for  particular  subsets  of  the  sample 
(Table  7)  and  correlations  (Table  8)  are  provided  here.  They 
will  be  referred  to  as  the  discussion  proceeds.  Regression 
results  will  be  presented  in  each  section  that  follows. 
1.   Random  Walk  Model  Bias  (BIASRW) 

Table  9  provides  the  results  of  the  multiple 
regression  analysis  for  BIASRW.  The  results  indicate  that 
approximately  33%  of  the  variation  in  random  walk  model  bias 
was  explained  by  the  nine  independent  variables.  However, 
only  two  variables — LRATE  and  FUTUPROD — were  significant  in 
explaining  variations  in  the  bias  of  random  walk  model  cost 
predictions.  The  relationships  between  random  walk  model  bias 
and  each  of  these  variables  are  depicted  in  Figure  1. 
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TABLE  7 


RELATIONSHIP  BETWEEN  MODEL  BIAS  AND 
SIGNIFICANT  EXPLANATORY  VARIABLES 


MODEL/ 

INDEPENDENT 

VARIABLE 


MEAN  BIAS  FOR  EACH  LEVEL 


LOW* 

MEDIUM5 

HIGH6 

RANDOM  WALK: 

LRATE 

0.052212 

0.027690 

0.074813 

FUTUPROD 

-0.064090 

0.024403 

0. 199371 

LEARNING 
CURVE: 

BURDEN 

0.024573 

-0.082520 

0. 111077 

LRATE 

-0.151880 

-0.054310 

0.219654 

BEGTREND 

0. 194725 

-0.056370 

-0. 122000 

ENDTREND 

-0.039350 

-0.013680 

0.032102 

FUTUPROD 

-0.170490 

-0.048520 

0.235615 

FIXED- 
VARIABLE 

FUTUPROD 

0.907445 

0.267981 

0.060342 

4  Identifies  variable  values  from  the  first  quartile  of  the 
variable's  distribution. 

Identifies  variable  values  from  the  second  and  third 
quartiles  of  the  variable's  distribution. 

6  Identifies  variable  values  from  the  fourth  quartile  of  the 
variable's  distribution. 
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TABLE  8 


VARIABLES 


VARIABLES 


CORRELATION  ANALYSIS  OF  INDEPENDENT 
VARIABLES  AND  MODEL  BIAS 


SPEARMAN  CORRELATION  COEFFICIENTS 
MODEL  BIAS 


BIASRW 

BIASLC 

BIASFV 

BIASBE 

PLOTPNTS 

-0.17974 

-0.05108 

0.10070 

0.01313 

MOD 

0.07265 

0.32932* 

0.00535 

0.15803 

BURDEN 

0.03855 

-0.15101 

-0.03751 

-0.08670 

CVAR 

0.09104 

-0.31316* 

0.13307 

-0.13021 

LRATE 

-0.07992 

0.34364* 

-0.25810* 

0.07666 

RATEVAR 

0. 10188 

-0. 06053 

0.02671 

-0. 01795 

BEGTREND 

0.00121 

-0.28616* 

0.27543* 

0.05199 

ENDTREND 

0.13639 

-0.02809 

0.10935 

0. 00670 

FUTUPROD 

0.42949* 

0.34583 

-0.28896* 

-0.11105 

PEARSON  CORRELATION  COEFFICIENTS 
MODEL  BIAS 


BIASRW 

BIASLC 

BIASFV 

BIASBE 

PLOTPNTS 

-0. 18426 

-0.10901 

0.08183 

-0.04182 

MOD 

0.10132 

0.26970* 

-0. 03685 

0.14173 

BURDEN 

0.21469 

0.29738* 

0.02674 

-0. 07563 

CVAR 

0.03327 

-0.09775 

-0.02379 

-0. 11590 

LRATE 

0.21866 

0.68339* 

-0.20121 

-0.01375 

RATEVAR 

0.04942 

-0.02878 

-0.13466 

-0. 12115 

BEGTREND 

-0.09221 

-0.35540* 

0.13413 

0.02868 

ENDTREND 

0.08477 

0. 07822 

0.03699 

-0.02651 

FUTUPROD 

0.46612* 

0.31543* 

-0.57893* 

-0. 18493 

*  Indicates   a  <.   0.01 
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TABLE    9 


MULTIVARIATE    ANALYSIS    OF    RANDOM    WALK    MODEL    BIAS 


Analysis    of   Variance 


Source 

Sum  of 
DF       Squares        S 

Mean 
quare 

F  Value 

Prob>F 

Model 
Error 
C  Total 

9       1. 

96       2. 

105      4. 

62871       0. 
53963       0. 
21833 

18097 
02698 

6 

709 

0.0001 

Root  MSE 
Dep  Mean 
C.V. 

0.16424 

0.05042 

325.76890 

R-square 
Adj  R-sq 

0.3361 
0.3285 

Parameter  Estimat 

es 

Variable   DF 

Parameter 
Estimate 

Standard 
Error 

T  for  HO: 
Parameter=0 

Pro 

b  : 

>  IT| 

INTERCEP    1 
BURDEN     1 
CVAR       1 
LRATE       1 
RATEVAR     1 
BEGTREND    1 
ENDTREND    1 
FUTUPROD    1 
PLOTPNTS    1 
MOD         1 

-0.366140 
0.159379 

-0.034241 
0.447784 

-0.062376 
0.136147 
0.044053 
0.126599 

-0.016620 
0.048861 

0.15950058 
0.06505715 
0.14791824 
0.15651890 
0.12325481 
0.06080772 
0.02678170 
0.02233301 
0.00932812 
0.04634494 

-2.296 
2.450 

-0.231 
2.861 

-0.506 
2.239 
1.645 
5.669 

-1.782 
1.054 

0 
0 
0 
0 

0 
0 
0 
0 
0 
0 

0239 

0161 

8174 

0052  * 

614-0 

0275 

1033 

0001  * 

0780 

2944 

*  Indicates  a   <,  0.01 
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RANDOM  WALK  BIAS 


0.25 


0.2 


0.15 


0.1 


0.05 


-0.05 


■0.1 


LRATE 


FUTUPROD 


LOW  0  MEDIUM   DhIGH 


Relationship  Between  Random  Walk  Model 
Bias  and  Significant  Explanatory  Variables 


Figure    1 
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a.  LRATE 

LRATE  was  found  to  be  significant:  in  explaining 
variations  in  the  bias  of  random  walk  model  cost  predictions 
in  both  the  simple  and  multiple  regression  analyses.  The 
estimated  regression  coefficients  indicated  a  positive 
relationship  between  LRATE  and  BIASRW.  Computed  Spearman  and 
Pearson  correlations  were  inconclusive  with  regard  to  the 
nature  of  the  relationship.  Figure  1  indicates  that  the 
random  walk  model  bias  was  positive  for  all  levels  of  LRATE. 
However,  the  relationship  between  LRATE  and  BIASRW  appears 
counterintuitive.  One  would  expect  low  (high)  values  of  LRATE 
to  be  associated  with  high  (low)  values  of  bias.  In  fact,  a 
nonlinear  relationship  exists.  Bias  is  highest  (most 
positive)  when  the  learning  rate  is  at  extremes,  i.e.,  when 
the  learning  rate  is  steepest  or  most  shallow.  Bias  is  lowest 
when  the  learning  rate  is  in  the  middle  range,  i.e.,  where 
most  programs  likely  will  fall. 

b.  FUTUPROD 

In  addition  to  LRATE,  FUTUPROD  was  also  found  to  be 
significant  in  explaining  random  walk  bias  in  both  the  simple 
and  multiple  regression  analyses.  The  estimated  regression 
coefficient  was  positive  in  both  analyses  thus  indicating  a 
positive  relationship  between  FUTUPROD  and  BIASRW.  Computed 
Spearman  and  Pearson  correlation  coefficients  for  FUTUPROD  and 
BIASRW  were  both  moderately  positive  and  significant,  thereby 
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confirming  this  relationship.  The  results  show  that  when 
FUTUPROD  was  low  (high)  the  level  of  positive  bias  was  low 
(high) .  This  relationship  is  readily  apparent  in  Figure  1 
which  shows  clearly  that  when  FUTUPROD  was  low  (high)  ,  the 
random  walk  model  underestimated  (overestimated)  unit  costs. 
These  results  were  expected.  When  FUTUPROD  is  high  (low) ,  unit 
costs  are  normally  lower  (higher)  in  the  period  being  forecast 
because  of  decreasing  (increasing)  variable  costs  per  unit 
(learning  effect)  and  allocation  of  total  fixed  costs  over  a 
larger  production  volume. 

2.   Learning  Curve  Model  Bias  (BIASLC) 

Table  10  provides  the  results  of  the  multiple 
regression  analysis  for  BIASLC.  The  results  indicate  that 
approximately  73%  of  the  variation  in  learning  curve 
prediction  bias  was  explained  by  the  nine  independent 
variables.  Five  of  the  independent  variables  included  in  the 
multiple  regression  analysis  were  significant  in  explaining 
variations  in  the  level  of  bias  of  learning  curve  model  cost 
predictions.  These  variables  were:  BURDEN,  LRATE,  BEGTREND, 
ENDTREND,  and  FUTUPROD.  In  general,  the  simple  regression 
results  agreed  with  the  multiple  regression  results  in  terms 
of  the  significance  of  these  variables. 
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TABLE    10 

MULTIVARIATE    ANALYSIS    OF    LEARNING    CURVE    MODEL    BIAS 

Analysis    of   Variance 


Source 

Sum  of 
DF       Squares        S 

Mean 
quare 

F  Va 

lue 

Prob>F 

Model 
Error 
C  Total 

9      12. 
96       4. 

105      17. 

92818      1. 
29728       0. 
22546 

43646 

04476 

32. 

090 

0.0001 

Root  MSE 
Dep  Mean 
C.V. 

0.21157 

-0.00776 

-2727 .49365 

R-square 
Adj  R-sq 

0.7505 
0.7271 

Parameter  Estimat 

es 

Variable   DF 

Parameter 
Estimate 

Standard 
Error 

T  for  HO: 
Parameter=0 

Pro 

b  : 

>  ITI 

INTERCEP    1 
BURDEN     1 
CVAR       1 
LRATE      1 
RATEVAR     1 
BEGTREND    1 
ENDTREND    1 
FUTUPROD    1 
PLOTPNTS    1 
MOD         1 

-2.325371 
0.372917 
0.127763 
2.316325 

-0.204656 
0.283523 
0.175378 
0.122591 

-0.011255 
0.137628 

0.20546621 
0.08380563 
0.19054602 
0.20162526 
0.15877497 
0.07833157 
0.03449978 
0.02876904 
0.01201634 
0.05970084 

-11.318 

4.450 

0.671 

11.488 

-1  .289 

3.620 

5.083 

4.261 

-0.937 

2.305 

0 
0 
0 
0 
0 
0 
0 
0 
0 
0 

0001 
0001  * 
5041 
0001  * 
2005 
0005  * 
0001  * 
0001  * 
3513 
0233 

*  Indicates  a   <,  0.01 
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However,  there  were  two  noteworthy  differences:  (1)  ENDTREND 
was  significant  in  the  multiple  regression  analysis  but  was 
not  significant  in  the  simple  regression  analysis;  and  (2)  MOD 
was  significant  in  the  simple  regression  analysis  but  was  not 
significant  in  the  multiple  regression  analysis.  These  two 
findings  will  be  discussed  in  detail  later  in  this  section. 
The  following  paragraphs  discuss  the  results  of  the  regression 
analyses  for  each  of  the  variables  identified  as  being 
significant.  The  relationships  between  the  level  of  each  of 
the  significant  independent  variables  and  model  bias  are 
depicted  in  Figure  2 . 
a.  BURDEN 

BURDEN  was  found  to  be  significant  in  explaining 
variations  in  model  bias  in  both  the  multiple  and  simple 
regression  analyses.  The  estimated  multiple  regression 
coefficient  indicates  there  was  a  positive  relationship 
between  burden  and  prediction  bias  for  the  learning  curve. 
This  result  was  confirmed  by  the  significant  but  relatively 
weak,  positive  Pearson  correlation  (See  Table  3)  .  These 
findings  indicate  that  when  the  proportion  of  total  cost  made 
up  of  fixed  costs  was  high  (low),  the  level  of  positive  bias 
was  high  (low).  Figure  2  graphically  depicts  the  relationship 
between  learning  curve  model  bias  and  BURDEN.  The  graph  shows 
that  low  (high)  positive  bias  was  indeed  associated  with 
low  (high)  levels  of  BURDEN. 
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However,  the  graph  also  indicates  that  negative  bias  occurred 
at  medium  BURDEN  levels.  In  other  words,  when  the  proportion 
of  total  cost  made  up  of  fixed  costs  was  low,  the  learning 
curve  model  slightly  overestimated  unit  costs.  In  addition, 
when  the  proportion  of  total  cost  made  up  of  fixed  costs  was 
at  a  medium  or  moderate  level,  the  learning  curve  model 
underestimated  unit  costs.  Finally,  when  the  proportion  of 
total  cost  made  up  of  fixed  costs  was  high,  the  learning  curve 
overestimated  unit  costs  by  a  moderately  large  amount.  This 
behavior  confirms  the  finding  in  the  earlier  study  by  Moses 
[Ref.  5],   Moses  found  that: 


Negative  bias  consistently  increases  with  increases  in 
fixed  cost  burden — up  to  a  point — then  negative  bias 
decreases  with  further  increases  in  burden.  The  turn 
around  point  for  all  observations  is  when  burden  is  50%. 
[Ref.  5:p.  27] 


Moses  attributes  this  behavior  to  the  fact  that  when  BURDEN  is 
0%  all  costs  are  variable  and  subject  to  learning.  In 
addition,  when  BURDEN  is  100%,  all  costs  are  fixed  and  are  not 
subject  to  learning.  Under  these  circumstances  the  learning 
curve  model  correctly  specifies  the  "true11  underlying  cost 
function  and  no  bias  will  result.  According  to  Moses,  bias 
results  only  when  costs — some  subject  to  learning  and  some 
not — are  combined.  [Ref.  5:p.  28] 
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b.    LRATE 

LRATE  was  found  10  be  significant  in  both  the 
simple  and  multiple  regression  analyses.  Moreover,  LRATE  was 
the  most  important  variable  in  terms  of  ability  to  explain 
variations  in  the  bias  of  learning  curve  model  predictions. 
This  is  evidenced  by  the  fact  that  its  t-value  far  exceeded 
the  t-values  for  the  other  explanatory  variables.  The 
positive  multiple  regression  coefficient  indicates  there  was 
a  positive  relationship  between  LRATE  and  BIASLC.  This 
relationship  was  confirmed  by  the  computed  Spearman  and 
Pearson  correlation  coefficients  (See  Table  8) .  The  Pearson 
correlation  coefficient  was  particularly  high  and  reflected  a 
relatively  strong,  positive,  linear  correlation  between  LRATE 
and  BIASLC.  The  results  indicate  that  when  LRATE  was  low 
(i.e.,  a  high  level  of  learning  was  occurring)  the  level  of 
positive  bias  was  low;  and,  when  LRATE  was  high  (i.e.,  a  low 
level  of  learning  was  occurring)  the  level  of  positive  bias 
was  high.  Figure  2  confirms  this  relationship.  When  the 
level  of  learning  was  high  (i.e.,  the  LRATE  was  low),  the 
learning  curve  model  greatly  underestimated  unit  costs.  When 
the  level  of  learning  was  moderately  high  (i.e.,  LRATE  was 
medium) ,  the  learning  curve  model  underestimated  unit  costs  by 
a  relatively  small  amount.  Finally,  when  the  level  of 
learning  was  low  (i.e.,  LRATE  was  high)  the  learning  curve 
model  greatly  overestimated  unit  costs. 
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The  findings  concerning  the  relationship  between 
LRATE  and  BIASLC  described  above  differ  markedly  from  the 
results  presented  in  the  Moses  study  [Ref.  5].  Moses  found 
that  learning  rate  was  not  significant  in  its  ability  to 
explain  variations  in  learning  curve  bias.  However,  the 
results  of  the  current  study  indicate  that  learning  rate  is 
extremely  important  in  terms  of  its  ability  to  explain  model 
bias.  This  suggests  that  the  traditional  learning  curve  model 
does  not  adequately  specify  the  affect  of  learning  on  variable 
costs. 

c.  BEGTREND 

BEGTREND  was  found  to  be  significant  in  both  the 
simple  and  multiple  regression  analyses.  Nevertheless,  the 
computed  regression  coefficients  provided  conflicting 
information  about  the  relationship  between  past  production 
trend  (measured  by  BEGTREND)  and  BIASLC.  The  simple 
regression  results  (See  Table  11)  indicated  a  negative 
relationship  while  the  multiple  regression  results  indicated 
a  positive  relationship. 

Computed  correlation  coefficients  for  BEGTREND  and 
BIASLC  were  examined  to  provide  another  look  at  the  nature  of 
the  relationship.  Both  Spearman  and  Pearson  correlation 
coefficients  reflected  the  existence  of  a  moderately  weak, 
negative  relationship  between  BEGTREND  and  BIASLC. 
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TABLE  11 


UNIVARIATE  ANALYSIS  OF  LEARNING  CURVE  MODEL  BIAS 

(BEGTREND) 


Analysis    of    Variance 

Sum    of  Mean 

Source  DF  Squares  Square  F    Value  Prob>F 

Model  1  3.55873  3.55873  27.081  0.0001 

Error  104  13.66673  0.13141 

C    Total  105  17 .22546 

Root    MSE  0.36251  R-square  0.2066 

Dep    Mean  -0.00776  Adj    R-sq  0.1990 

C.V.  -4673.24228 

Parameter    Estimates 

•  Parameter  Standard  T    for    HO: 

Variable      DF  Estimate  Error         Parameter=0  Prob    >    |T| 

INTERCEP        1  0.191679  0.05204281  3.683  0.0004 

BEGTREND        1  -0.381017  0.07321702  -5.204  0.0001    * 


*  Indicates  a   ^  0.01 
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This  indicates  that  when  the  past  production  trend  was 
decreasing  (increasing),  (i.e.,  the  initial  production  volume 
per  period  was  above  (below)  the  average  production  volume  per 
period),  the  level  of  positive  bias  was  high  (low).  The 
relationship  between  BEGTREND  and  BIASLC  is  depicted  in  Figure 
2 .  The  graph  shows  that  when  the  past  production  trend  was 
decreasing  (i.e.,  BEGTREND  was  negative)  learning  curve  bias 
was  highly  positive.  In  addition,  when  past  production  trend 
was  relatively  stable  (i.e.,  BEGTREND  was  near  zero)  learning 
curve  bias  was  negative.  Finally,  the  graph  shows  that  when 
the  past  production  level  was  increasing  (i.e.,  BEGTREND  was 
positive)  learning  curve  bias  was  highly  negative. 

In  general,  the  results  indicate  that  when 
production  volume  at  the  beginning  of  a  series  of  production 
lots  (i.e.,  at  the  start  of  a  program)  starts  off  low  and 
subsequently  builds  upward  to  a  higher  volume,  the  learning 
curve  model  has  a  strong  tendency  to  underestimate  future 
costs.  In  contrast,  when  initial  production  volume  starts  off 
high,  the  learning  curve  is  biased  toward  overestimating 
future  costs. 

The  relationship  between  BEGTREND  and  BIASLC 
described  above  is  consistent  with  the  results  of  Moses'  study 
in  terms  of  the  nature  of  the  relationship.  However,  Moses 
found  that  learning  curve  bias  was  negative  for  all  production 
trends.  This  phenomenon  was  not  observed  in  the  current 
study. 
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In  an  effort  to  determine  the  source  of  the 
conflicting  results  from  "Che  regression  analyses,  correlations 
between  BEGTREND  and  the  other  independent  variables  were 
examined.  Correlations  between  the  independent  variables  are 
provided  in  Table  12.  The  correlations  indicate  significant, 
relatively  strong  relationships  existed  between  BEGTREND  and 
CVAR,  BEGTREND  and  LRATE,  and  BEGTREND  and  RATEVAR. 
Consequently,  there  is  a  strong  possibility  that 
multicollinearity  existed  between  these  variables.  Such  a 
condition  could  have  contributed  to  the  conflicting  regression 
results . 

Another  possible  factor  which  could  have  influenced 
the  results  is  that  there  may  have  been  interactions  between 
BEGTREND  and  the  other  variables  in  the  multiple  regression 
analysis.  Interactions  between  BURDEN  and  past  production 
trend  were  identified  in  the  Moses  study  and  were  found  to  be 
significant  in  explaining  variations  in  BIASLC. 
d.    EN DT REND 

ENDTREND  was  found  to  be  significant  in  explaining 
model  bias  in  the  multiple  regression  analysis  only.  This 
result  was  thought  to  be  largely  the  result  of  interactions 
between  ENDTREND  and  the  other  independent  variables.  The 
estimated  multiple  correlation  coefficient  indicated  a 
positive  relationship  existed  between  ENDTREND  and  BIASLC. 
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However,  this  relationship  could  not  be  confirmed  by  examining 
rhe  computed  correlation  coefficients.  Neither  the  Pearson 
correlation  coefficient  not  the  Spearman  correlation 
coefficient  was  significant  for  the  relationship  between 
ENDTREND  and  BIASLC. 

The  positive  relationship  between  ENDTREND  and 
BIASLC  implies  that  when  the  past  production  trend  (ENDTREND) 
was  decreasing  (increasing)  (i.e.,  the  guantity  produced  in 
the  most  recent  period  was  less  than  (greater  than)  the 
average  guantity  produced  per  period) ,  the  level  of  positive 
bias  was  low  (high) .  This  relationship  between  ENDTREND  and 
BIASLC  is  shown  in  Figure  2.  The  graph  shows  that  when 
ENDTREND  was  low  (decreasing  production  trend)  BIASLC  was 
negative.  In  addition,  when  ENDTREND  was  medium  (relatively 
stable  production  trend)  BIASLC  was  less  negative.  Finally, 
the  graph  shows  that  when  ENDTREND  was  high  (increasing 
production  trend)  BIASLC  was  positive. 

In  the  most  general  terms,  the  results  indicate 
that  when  production  volume  at  the  end  of  a  series  of 
production  lots  is  declining,  the  learning  curve  model  has  a 
tendency  to  underestimate  future  costs.  Similarly,  when 
production  volume  at  the  end  of  a  series  of  production  lots  is 
increasing,  the  learning  curve  model  has  a  tendency  to 
overestimate  future  costs.  Note,  however,  these  tendencies 
are  mild,  as  indicated  by  the  small  magnitude  of  the  effects 
in  Figure  2 . 
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e.  FUTUPROD 

FUTUPROD  was  found  to  be  significant:  in  explaining 
model  bias  in  both  the  simple  and  multiple  regression 
analyses.  The  estimated  regression  coefficient  was  positive 
in  both  analyses  thus  indicating  a  positive  relationship 
between  FUTUPROD  and  BIASLC.  Computed  Spearman  and  Pearson 
correlation  coefficients  for  FUTUPROD  and  BIASLC  were  both 
weakly  positive  and  significant,  thereby  confirming  this 
relationship.  The  results  indicate  that  when  FUTUPROD  was 
low  (high)  the  level  of  bias  was  low  (high) .  This 
relationship  is  readily  apparent  in  Figure  2.  The  graph  shows 
that  when  FUTUPROD  was  low,  BIASLC  was  highly  negative.  When 
FUTUPROD  was  medium  (i.e.,  the  production  level  for  the  next 
period  was  comparable  with  the  production  level  for  the  last 
period)  BIASLC  was  moderately  negative.  Finally,  when 
FUTUPROD  was  high,  BIASLC  was  highly  positive. 

The  relationship  between  FUTUPROD  and  BIASLC 
observed  in  this  study  coincides  almost  exactly  with  the 
relationship  described  in  the  Moses  study  [Ref.  5].  As 
expressed  by  Moses,  this  relationship  should  be  expected. 

Higher  (lower)  future  production  will  result  in  lower 
(higher)  fixed  cost,  and  total  cost,  per  unit,  creating  a 
tendency  toward  positive  (negative)  bias  for  any  cost 
estimate.  [Ref.  5:p.  22] 

In  general,  these  findings  indicate  that  the 
learning  curve  model  tends  to  underestimate  unit  costs  when  it 


86 


is  used  to  predict  costs  for  periods  in  which  production 
volume  is  cut  back.    Conversely,  the  learning  curve  model 
tends   to   overestimate   unit   costs   for   periods   in  which 
production  volume  is  increased. 
f.    MOD 

MOD  was  identified  as  being  significant  in 
explaining  variations  in  bias  in  the  simple  regression 
analysis  (See  Table  13);  however,  MOD  was  not  significant  in 
the  multiple  regression  analysis.  Both  Spearman  and  Pearson 
correlation  coefficients  were  examined.  The  correlation 
coefficients  indicated  a  significant,  weakly  positive 
correlation  between  MOD  and  BIASLC.  This  suggests  that  the 
learning  curve  model  has  a  lesser  tendency  to  underestimate 
unit  costs  when  used  in  predicting  costs  for  modification  type 
programs.  Examination  of  the  correlations  between  MOD  and  the 
other  explanatory  variables  revealed  there  was  a  significant, 
moderately  strong  positive  correlation  between  MOD  and  LRATE. 
Consequently,  it  is  likely  that  multicollinearity  between  MOD 
and  LRATE  was  the  source  of  the  lack  of  significance  for  MOD 
in  the  multiple  regression  analysis.  This  positive 
correlation  between  MOD  and  LRATE  seems  reasonable.  One  would 
expect  modification  programs  to  exhibit  less  learning  than 
programs  involving  entirely  new  designs. 
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TABLE  13 


UNIVARIATE  ANALYSIS  OF  LEARNING  CURVE  MODEL  BIAS 

(MOD) 


Analysis    of   Variance 

Sum    of  Mean 

Source  DF  Squares  Square  F    Value  Prob>F 

Model  1  1.19852  1.19852  7.777  0.0063 

Error  104  16.02694  0.15411 

C    Total  105  17.22546 

Root    MSE  0.39256  R-square  0.0696 

Dep   Mean  -0.00776  Adj    R-sq  0.0606 

C.V.  -5060.70855 

Parameter    Estimates 

Parameter  Standard  T    for    HO: 

Variable      DF  Estimate  Error         Parameter=0  Prob    >    |T| 

INTERCEP        1  -0.156032  0.06542706  -2.385  0.0189 

MOD  1  0.224530  0.08051213  2.789  0.0063    * 


*  Indicates  a   <.  0.01 
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3.   Fixed-Variable  Model  Bias  (BIASFV) 

Table  14  provides  the  results  of  the  multiple 
regression  analysis  for  BIASFV.  The  results  indicated  thar 
approximately  37%  of  the  variation  in  fixed-variable  model 
bias  was  explained  by  the  nine  independent  variables. 
However,  only  one  of  these  variables — FUTUPROD — was 
significant  in  explaining  variations  in  the  bias  of  fixed- 
variable  model  cost  predictions.  FUTUPROD  was  also  found  to 
be  significant  in  the  simple  regression  analysis.  The 
estimated  multiple  regression  coefficient  indicates  a  negative 
relationship  existed  between  FUTUPROD  and  BIASFV.  This 
relationship  is  confirmed  by  the  computed  Spearman  and  Pearson 
correlation  coefficients  for  FUTUPROD  and  BIASFV.  Both 
correlation  coefficients  indicate  the  existence  of  a 
significant  negative  correlation.  Moreover,  the  relatively 
large  negative  Pearson  correlation  coefficient  indicates  that 
the  correlation  between  FUTUPROD  and  BIASFV  was  fairly  linear. 
The  relationship  between  fixed-variable  model  bias  and 
FUTUPROD  is  depicted  in  Figure  3 . 

The  graph  confirms  that  when  the  level  of  production 
in  the  next  period  was  high  (low)  relative  to  the  most  recent 
period,  the  level  of  positive  bias  was  low  (high) .  In  short, 
the  fixed-variable  model  has  a  tendency  to  overestimate  future 
costs.  This  tendency  is  greatest  when  the  model  is  used  to 
predict  costs  for  periods  in  which  cutbacks  in  production 
volume  occur. 
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TABLE    14 

MULTIVARIATE    ANALYSIS    OF    FIXED-VARIABLE    MODEL    BIAS 


Analysis    of    Variance 


Source 

Sum  of 
DF       Squares        S 

Mean 
quare 

F  Value 

Prob>F 

Model 

Error 
C  Total 

9      42. 

96      57. 
105     100. 

65513       4. 
98486       0. 
63998 

73946 
60401 

7 

847 

0.0001 

Root  MSE 
Dep  Mean 
C.V. 

0.77718 

0.42647 

182.23765 

R-square 
Adj  R-sq 

0.4238 
0.3698 

Parameter  Estimat 

es 

Variable   DF 

'  Parameter 
Estimate 

Standard 
Error 

T  for  HO: 
Parameter=0 

Pro 

b  - 

>  ITI 

INTERCEP    1 
BURDEN      1 
CVAR        1 
LRATE       1 
RATEVAR     1 
BEGTREND    1 
ENDTREND    1 
FUTUPROD    1 
PLOTPNTS    1 
MOD         1 

1.627186 

0.351109 

-0.833763 

-1.572097 

-0.016944 

-0.224974 

0.010113 

-0.751819 

0.049066 

0.094664 

0.75474637 
0.30784621 
0.69993949 
0.74063728 
0.58323376 
0.28773817 
0.12672928 
0.10567336 
0.04414006 
0.21930124 

2.156 

1.141 

-1.191 

-2.123 

-0.029 

-0.782 

0.080 

-7.114 

1.112 

0.432 

0 
0 
0 
0 
0 
0 
0 
0 
0 
0 

.0336 
2569 
2365 
0364 
9769 
4362 
9366 
0001  * 
2691 
6670 

*  Indicates  <x   <  0.01 


90 


FIXED-VARIABLE   BIAS 


0.8 


0.6 


0.4 


0.2 


FUTUPROD 


■  low  H  medium  Dhigh 

Relationship  Between  Fixed-Variable  Model 
Bias  and  Significant  Explanatory  Variable 

Figure    3 


91 


4.   Bemis  Rate  Adjustment  Model  Bias  (BIASBE) 

Table  15  provides  the  results  of  the  multiple 
regression  analysis  for  BIASBE.  Only  approximately  2%  of  the 
variation  in  Bemis  model  prediction  bias  was  explained  by  the 
nine  independent  variables.  Moreover,  none  of  the  variables 
were  found  to  be  significant  in  either  the  simple  regression 
or  multiple  regression  analyses.  This  means  that  the  Bemis 
model  was  successful  in  accounting  for  the  influences  of  these 
variables.  The  findings  concerning  Bemis  model  performance 
support  the  conclusions  drawn  in  the  earlier  Moses  study. 
Moses  found  that  the  overall  mean  bias  for  all  cost 
predictions  made  with  the  Bemis  rate  adjustment  model  was 
-0.0016.  As  a  result,  he  concluded  that,  on  average,  the  rate 
adjustment  model  exhibits  no  bias.  In  addition,  Moses 
observed  that  the  absence  of  bias  was  evident  for  all 
treatments  across  all  variables  of  interest.  No  significant 
main  effects  were  observed  in  the  ANOVA  results.  [Ref .  5:p.24] 
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TABLE    15 


MULTIVARIATE    ANALYSIS    OF    BEMIS    MODEL    BIAS 


Analysis    of    Variance 


Source 

Sum  of 
DF      Squares       S 

Mean 
quare 

F  Mc 

ilue 

Prob>F 

Model 
Error 
C  Total 

9       1. 

96      11. 

105      13. 

33915      0. 
96836       0. 
30751 

14879 
12467 

1 

194 

0.3081 

Root  MSE 
Dep  Mean 
C.V. 

0.35309 

0.03091 

1142.35596 

R-square 
Adj  R-sq 

0.1006 
0.0163 

Parameter  Estimat 

es 

Variable   DF 

•  Parameter 

Estimate 

Standard 
Error 

T  for  HO: 
Parameter=0 

Pro 

b  >  |T| 

INTERCEP    1 
BURDEN      1 
CVAR       1 
LRATE      1 
RATEVAR     1 
BEGTREND    1 
ENDTREND    1 
FUTUPROD    1 
PLOTPNTS    1 
MOD         1 

0.097655 
-0.055001 

0.161554 
-0.040905 
-0.424442 

0.090834 

0.044540 
-0.075044 
-0.008266 

0.192540 

0.34289489 
0.13986008 
0.31799513 
0.33648487 
0.26497361 
0.13072464 
0.05757540 
0.04801158 
0.02005362 
0.09963251 

0.285 
-0.393 

0.508 
-0.122 
-1.602 

0.695 

0.774 
-1.563 
-0.412 

1.933 

0 
0 
0 

0 
0 
0 
0 
0 
0 
0 

7764 
6950 
6126 
9035 
1125 
4888 
4411 
1213 
6811 
.0562 
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E.   SUMMARY 

This  chapter  has  described  the  results  of  statistical 
analyses  of  the  bias  associated  with  unit  cost  predictions 
obtained  using  the  random  walk,  traditional  learning  curve, 
fixed-variable  and  Bemis  production  rate  adjustment  models. 
The  discussion  began  with  a  description  of  the  statistical 
procedures  used  in  conducting  the  study.  This  was  followed  by 
a  presentation  of  the  findings  with  respect  to  overall 
performance  of  the  four  cost  progress  models  and  an 
examination  of  the  factors  thought  to  be  useful  in  explaining 
variations  in  model  performance  (bias) . 

Median  bias  values  for  the  four  models  indicated  that  the 
traditional  learning  curve  model  and  the  Bemis  model  tend  to 
underestimate  the  unit  costs  of  weapons  systems  while  the 
random  walk  and  fixed-variable  models  tend  to  overestimate 
unit  costs.  In  addition,  correlation  analyses  of  the  cost 
prediction  bias  associated  with  each  of  the  models  reflected 
a  significant  positive  correlation  between  predicted  unit  cost 
bias  for  all  paired  model  combinations.  ANOVA  and  regression 
analyses  were  conducted  to  determine  the  significance  of  the 
four  demographic  variables  and  eight  condition  variables 
included  in  the  study.  Only  three  of  the  demographic 
variables — MISSION,  MOD  and  TYPE  x  MOD  (interaction 
variable) — were  significant  in  terms  of  their  ability  to 
explain  variations  in  the  level  of  model  prediction  bias. 
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Moreover,  the  significance  of  these  variables  was  limited  to 
the  learning  curve  model. 

Findings  concerning  the  significance  of  the  eight 
condition  variables  revealed  that  the  utility  of  these 
variables  in  explaining  model  cost  prediction  bias  varied 
widely  among  the  models.  Only  two  condition  variables — LRATE 
and  FUTUPROD — were  significant  in  explaining  variations  in  the 
bias  of  random  walk  cost  predictions.  Conversely,  five 
condition  variables — BURDEN,  LRATE,  BEGTREND,  ENDTREND  and 
FUTUPROD — were  significant  in  explaining  variations  in  the 
bias  of  learning  curve  model  cost  predictions.  FUTUPROD  was 
the  only  condition  variable  that  was  significant  in  explaining 
variations  in  the  bias  of  fixed-variable  model  cost 
predictions.  Finally,  none  of  the  condition  variables  was 
significant  in  explaining  variations  in  the  bias  of  Bemis 
model  cost  predictions. 

The  results  of  the  statistical  analyses  of  learning  curve 
cost  prediction  bias  differed  somewhat  from  those  obtained  by 
Moses  [Ref.  5],  particularly  with  respect  to  the  significance 
of  LRATE  and  PLOTPNTS  in  explaining  variations  in  cost 
prediction  bias.  However,  results  for  the  Bemis  production 
rate  adjustment  model  strongly  supported  the  findings 
presented  by  Moses.  Chapter  V  will  summarize  the  major 
findings  from  the  current  study  and  will  suggest  directions 
for  future  research. 
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V.   SUMMARY 

A.   FINDINGS 

The  primary  objective  of  this  study  was  to  determine  the 
bias  of  selected  cost  progress  models  when  predicting  the 
future  unit  cost  of  weapons  systems  acquired  through  a 
continuing  acquisition  program.  In  addition,  the  research 
sought  to  answer  the  following  questions: 


1.  Are  the  various  cost  progress  models  comparable  in  terms 
of  bias? 

2.  Do  particular  models  result  in  less  biased  estimates 
under  certain  conditions? 

3 .  What  are  those  conditions  that  affect  the  performance  of 
the  models? 

4.  Can  guidelines  be  established  for  determining  when  (under 
what  conditions  or  circumstances)  it  is  most  appropriate 
to  use  a  particular  model  type? 


This  chapter  will  address  each  of  these  areas  by  summarizing 
the  results  of  the  study. 

Overall  findings  with  respect  to  the  bias  of  unit  cost 
predictions  obtained  with  the  random  walk,  traditional 
learning  curve,  fixed-variable,  and  Bemis  production  rate 
adjustment  models  were  as  follows: 


1.  On  average,  the  learning  curve  model  underestimated  unit 
costs  by  approximately  6.1%. 
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2.  On  average,  the  Bemis  model  underestimated  unit  costs  by 
approximately  1.3%. 

3.  On  average,  the  random  walk  model  overestimated  unit 
costs  by  1.6%. 

4.  On  average,  the  fixed-variable  model  overestimated  unit 
costs  by  18.8%. 


These  findings  indicate  that  the  four  models  do  indeed  differ 
in  terms  of  the  direction  and  magnitude  of  cost  prediction 
bias . 

The  influence  of  various  factors  on  model  performance  was 
examined  by  evaluating  the  significance  of  four  demographic 
variables  and  eight  condition  variables  in  explaining 
variations  in  the  bias  of  unit  cost  predictions.  Findings 
were  that  the  utility  of  these  variables  in  explaining  model 
cost  prediction  bias  varied  widely  among  the  models.  In 
particular,  the  following  relationships  were  observed: 


1.  Random  walk  model  cost  prediction  bias  is  influenced  by 
two  factors — the  learning  rate  associated  with  the 
production  process,  and  the  production  level  for  the 
future  period  relative  to  the  most  recent  period. 

2.  When  the  level  of  learning  is  high,  the  random  walk  model 
overestimates  unit  costs  by  approximately  5.2%.  At 
moderately  high  learning  levels,  the  random  walk  model 
overestimates  unit  costs  by  only  approximately  2.8%. 
However,  when  the  level  of  learning  is  low,  the  random 
walk  model  overestimates  unit  cost  by  approximately  7.5%. 

3.  The  higher  the  production  level  in  a  future  period  (the 
period  for  which  unit  costs  are  being  forecast)  relative 
to  the  most  recent  production  period,  the  more  the 
random  walk  model  tends  to  overestimate  future  cost. 
When  the  future  production  level  is  significantly  lower 
than  the  level  in  the  most  recent  production  period,  unit 
costs  are  underestimated  by  approximately  6.4%.  When  the 
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future  production  level  is  comparable  with  the  most 
recent  period's  production  level,  unit  costs  are 
overestimated  by  approximately  2.4%.  Finally,  when  the 
future  production  level  is  significantly  higher  than  the 
level  in  the  most  recent  production  period,  unit  costs 
are  overestimated  by  approximately  20.0%. 

Learning  curve  model  cost  prediction  bias  is  influenced 
by  the  following  factors:  the  percentage  of  total  cost 
made  up  of  fixed  costs,  the  level  of  learning  associated 
with  the  production  process,  the  past  production  trend, 
and  the  future  production  level. 

When  the  proportion  of  total  cost  made  up  of  fixed  costs 
varies,  learning  curve  model  bias  is  affected  as  follows. 
At  low  levels,  unit  costs  are  overestimated  by 
approximately  2.5%;  at  medium  levels,  unit  costs  are 
underestimated  by  approximately  8.3%;  at  high  levels  unit 
costs  are  overestimated  by  approximately  11.1%. 

The  higher  the  level  of  learning,  the  more  positive  the 
bias  of  learning  curve  model  unit  cost  estimates.  When 
the  level  of  learning  is  high,  costs  are  underestimated 
by  approximately  15.2%.  When  the  level  of  learning  is 
moderately  high,  costs  are  underestimated  by 
approximately  5.4%.  When  the  level  of  learning  is  low, 
costs  are  overestimated  by  approximately  22.0%. 

When  the  production  volume  per  period  is  decreasing  at 
the  beginning  of  a  series  of  production  lots,  the 
learning  curve  model  overestimates  unit  costs  by 
approximately  19.5%.  When  the  beginning  production  trend 
is  relatively  stable,  the  learning  curve  model 
underestimates  unit  costs  by  approximately  5.6%.  When 
the  production  volume  per  period  at  the  beginning  of  a 
series  of  production  lots  is  increasing,  the  learning 
curve  model  underestimates  unit  costs  by  approximately 
12.2%. 

When  the  production  volume  per  period  is  decreasing  at 
the  end  of  a  series  of  production  lots,  the  learning 
curve  model  underestimates  unit  costs  by  approximately 
3.9%.  When  the  ending  production  trend  is  relatively 
stable,  the  learning  curve  model  underestimates  unit 
costs  by  approximately  1.4%.  When  the  production  volume 
per  period  is  increasing  at  the  end  of  a  series  of 
production  lots,  the  learning  curve  model  will 
overestimate  unit  costs  by  approximately  3.2%. 
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9.  There  is  a  positive  relationship  between  the  level  of 
production  in  a  future  period  (relative  to  the  most 
recent  period)  and  learning  curve  model  cost  prediction 
bias.  When  the  level  of  production  in  a  future  period  is 
significantly  lower  than  the  level  of  production  in  the 
most  recent  period,  the  learning  curve  model 
underestimates  unit  costs  by  approximately  17.0%.  When 
the  future  production  level  is  comparable  with  the 
production  level  in  the  most  recent  period,  the  learning 
curve  model  underestimates  unit  costs  by  approximately 
4.9%.  Finally,  when  the  future  production  level  is 
significantly  higher  than  the  level  of  production  in  the 
most  recent  period,  the  learning  curve  model  will 
overestimate  unit  costs  by  approximately  23.6%. 

10.  Fixed-variable  model  cost  prediction  bias  is  influenced 
by  the  level  of  production  in  a  future  period.   There  is 
a  negative  relationship  between  the  level  of  production 
in  a  future  period  (relative  to  the  most  recent  period) 
and  fixed-variable  model  bias.   When  the  future 
production  level  is  significantly  lower,  the  fixed- 
variable  model  overestimates  unit  costs  by  approximately 
9.1%.   When  the  future  production  level  is  relatively 
stable,  the  fixed-variable  model  overestimates  unit  costs 
by  approximately  26.8%.   Finally,  when  the  future 
production  level  is  significantly  higher,  the  fixed- 
variable  model  overestimates  unit  costs  by  approximately 
6.0%. 

11.  Bemis  production  rate  adjustment  model  cost  prediction 
bias  is  not  significantly  related  to  any  of  the  variables 
included  in  the  study. 


The  selection  of  a  particular  cost  progress  model  for 
estimating  airframe  unit  costs  depends  primarily  on  the 
availability  of  required  data  and  the  cost  versus  the  benefits 
of  collecting  the  additional  data  required  to  employ  more 
sophisticated  cost  prediction  models.  The  findings  from  this 
study  indicate  that  the  learning  curve,  Bemis  and  random  walk 
models  all  produce  cost  predictions  which  have  low  biases. 
However,  the  models  differ  widely  in  terms  of  their 
susceptibility  to  vagaries  in  the  production  process.  Ceteris 
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paribus,  the  Bemis  model  is  superior  to  rhe  other  cost 
progress  models  because  it  not  only  exhibits  the  smallest  bias 
but  also  is  not  significantly  influenced  (in  terms  of  bias)  by 
variations  in  the  factors  considered  in  this  study.  Hence,  it 
provides  the  most  robust  and  consistent  cost  estimates. 
Conversely,  the  bias  of  unit  cost  predictions  obtained  with 
the  random  walk,  learning  curve,  and  fixed-variable  models  is 
significantly  influenced  by  variations  in  the  production 
process.  Consequently,  the  findings  presented  above  should  be 
considered  when  employing  these  less  sophisticated  models. 

B.   FUTURE  RESEARCH  OPPORTUNITIES 

The  results  obtained  in  the  study  suggest  other  potential 
directions  for  future  research: 


1.  The  current  study  could  be  extended  to  include  aircraft 
engines  and  missile  propulsion  systems  to  determine 
whether  the  findings  concerning  model  prediction  bias 
hold  when  the  models  are  used  to  predict  unit  costs  for 
these  systems. 

2 .  Further  studies  could  be  conducted  to  determine  the  exact 
nature  of  the  relationships  between  aircraft 
mission/missile  mission,  modification  status,  type  of 
system  (aircraft  or  missile)  and  learning  curve  model 
prediction  bias. 
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